Handwritten character recognition belongs to the image feature classification problem. To address the problems that convolutional neural network-based handwritten character feature extraction algorithms have complex structures, long training time and recognition accuracy of similar characters is much lower than the average recognition accuracy, we improve the VGG16 and propose a new character recognition algorithm, which effectively improves the recognition accuracy of similar characters. The improved model reduces the complexity and accelerates the model convergence by simplifying the original convolutional structure, adding Ghost convolution, and batch normalization layer. In addition, we introduce an attention mechanism and deformable convolution to solve the problem of false recognition caused by local feature differences and stroke deformation of similar characters. Experimental results on Chinese character datasets and self-built datasets show that our proposed approach achieves over 97% accuracy in general and 92.82% in similar characters. Compared with the traditional approaches, the training speed is faster, and the similar character recognition effect is improved. The model size is reduced by 80%, achieving lightweight and having important application value for improving the performance of character recognition.