A cup winding permanent magnet synchronous motor(PMSM) is proposed for the application of flywheel energy storage system(FESS) in wind power generation, which can effectively improve the efficiency of the motor/generator and reduce the height of the flywheel. Firstly, the structure of the whole flywheel system and the cup winding PMSM are given. Secondly, the preliminary design scheme and the main parameters of the motor are obtained, and the influence of parameters on the motor performance are analyzed by 3-D FEM. Then a new optimization strategy combining BP-ANN based on genetic algorithm(GA) and multi-objective PSO was proposed to optimize the motor design, and the results of optimized design are compared with the preliminary scheme. At last, preliminary experiments of the prototyped flywheel is conducted to verify the partial analysis and the excellent performance.