Accurate computation of protein-ligand binding free energy remains an elusive goal due to inherent difficulties involved in the accurate calculation of gas-phase protein-ligand interaction energy, the entropy, and the solvation energy. In this study, we explore the use of fragment quantum chemical calculations for improved accuracy in protein-ligand binding free energy calculations. The present work demonstrated that the gas-phase protein-ligand interaction energies can be accurately calculated by the molecular fractionation with conjugate caps method as verified by comparison with the full quantum calculations for several protein-ligand systems. The m06-2x/6-31+G* level of density functional theory calculation with basis set superposition error correction is found to give excellent protein-ligand interaction energies. The quantum calculated protein-ligand interaction energies are then combined with implicit solvation methods to obtain absolute binding free energies and the results are shown to be sensitive to the specific solvation models used. In particular, the accuracy of the quantum calculated binding free energies is significantly improved over that of the force field calculations using the same solvation models in terms of mean absolute errors. However, the correlation coefficients with respect to the experimental data do not show improvement over the corresponding result computed from the force field. Such result and the related analysis underscore the critical importance of solvation energies to the binding free energies and the need for developing new methods to calculate solvation energies more accurately in the future.