We perform protein structure prediction by combining a hybrid energy function, fragment assembly, and double optimization. In the hybrid energy function, all the backbone atoms are described explicitly, but the side-chain is modeled as a few interaction centers in order to reduce computational costs. We reduce the search space by using a fragment assembly method, where the local structure of the backbone is obtained from a structural database using similarity of sequence features, and only the global tertiary packing of fragments is determined by minimizing the energy. The structure with the minimum energy is obtained using double optimization, where a combination of backbone fragments with minimum energy is obtained using the conformational space annealing (CSA) method, and the optimal side-chains for a given backbone structure are obtained using simulated annealing. We show the feasibility of our method by performing test predictions on two proteins, 1bdd and 1e0l, that belong to distinct structural classes.