Machine learning has been widely adopted in various disciplines as they offer low-computational cost solutions to complex problems. Recently, deep learning-enabled methods for metasurface design have received increasing attention in the field of subwavelength electromagnetics. However, the previous metasurface design methods based on deep learning usually use huge datasets or complex networks to make deep neural networks achieve high prediction accuracy which results in more time for dataset establishment and network training. Here, we propose an expeditious and accurate scheme for designing phase-modulating dielectric metasurface through employing the transfer learning technology and genetic algorithm. The performance of the neural network is improved distinctly by migrating knowledge between real part and imaginary part spectrum-prediction tasks. Furthermore, the target meta-atoms can be optimized readily without increasing a large dataset through transfer learning. Finally, we design two deflectors and two metalenses as a proof-of-concept demonstration to validate the ability of our proposed approach. The scheme provides an efficient and promising design method for phase-modulating metasurface.