In recent years, Convolutional Neural Networks (CNN, CovNets) have made breakthrough achievements in image classification and face recognition and so on. The recognition rate and real-time performance of the current trajectory gesture recognition method cannot satisfy the needs of interacting with the intelligent teaching interface which is designed and implemented by us. In order to solve the problem of low recognition rate of the trajectory gesture and poor real-time performance, we proposed a method which the Convolutional Neural Networks is used to recognition the trajectory gesture. In this paper, we adopt Kinect sensor to capture the trajectory point of trajectory gesture. And then, the trajectory points are fitted the trajectory curve, and then it is standardized into an image of trajectory gesture. Experiments show that the recognition rate of trajectory gesture is 97.3%, and under the ordinary PC, the recognition time of each trajectory gesture is 20.69ms. The proposed method has well real-time performance.