Dual Multi-Task Network with Bridge-Temporal-Attention for Student Emotion Recognition via Classroom Video

Emotion recognition is one of the most significant technologies for building a smart education environment. The quantitative analysis of students' emotion in classroom is helpful to improve teaching effect. Though temporal features of emotion generation and disappearance process has been demonstrated to be of great benefit for emotion recognition, it has received little attention. Recently, a large of data has been accumulated in education, like classroom video. Considering the practical scenarios, it is more challenge to obtain the temporal information of video emotion. Therefore, this paper firstly uses the deep learning and Attention Mechanism methods to automatically modelling the temporal process. Besides, we make use of multitask learning to build a DMTN-BTA model for students' emotion recognition through classroom videos. While, temporal segment labels are unnecessary in the model. The method consists of the CNN for spatio-temporal features extraction on emotion recognition and temporal segmentation tasks and BLSTM-RNN with a novel ‘Bridge-Temporal-Attention’ for the emotion recognition task. Experiment results show that our model outperforms the prior single-task learning methods on the BNU-LSVED 2.0 and the current state-of-the-art methods on the official FABO.