Deep Feature Extraction and Attention Fusion for Multimodal Emotion Recognition

Recently, electroencephalogram (EEG)-based multimodal emotion recognition has emerged as one of the research hotspots in affective computing. However, the existing methods tend to ignore the interaction information between the EEG and other modal features. In this brief, we propose a novel model termed EEANet (EEG and eye movement Attention Network) to find the modal correlation at feature level. The DE feature and 31 eye movement features were extracted from the pre-processed EEG and eye movement signals, and then two feedforward encoders were used to capture the deep features, respectively. The interactive attention layer is applied to learn multi-modal complementary information and semantic-level context information. Finally, the multi-head self-attention mechanism allows the model to focus on the discriminative features for emotion classification. The model was verified on the SEED-IV dataset, and the results showed that the accuracy of emotion recognition was significantly improved with the EEANet, and the average accuracy of the four classifications was 92.26%.