Emotion recognition based on multiple physiological signals

Physiological signals can more realistically reflect human emotional states. To overcome the limitations imposed in single-modal emotion recognition, emotion recognition of multimodal physiological signals has received increasingly widespread attention. However, the original fusion models usually ignore the different distributions of multiple signals and how to capture complementary features from multimodal information effectively. This paper proposes an effective classification model for multimodal physiological signals to address the above issues based on modeling the heterogeneity and correlation between multimodal signals. First, differential entropy features are extracted from Electroencephalography (EEG) signals and peripheral physiological signals (PPS) such as Electrocardiographic (ECG) signals, Electromyographic (EMG) signals, and other physiological signals. Then, according to the different distributions and frequency characteristics of the acquired signals, the EEG signal features are made into a three-dimensional feature map and input to the neural network to extract the frequency spatial dimension features. Further temporal features are extracted from the peripheral physiological signals using a long and short-term memory network. Finally, the EEG and peripheral physiological signal features were fused and input to a multimodal long and short-term memory network to extract the association between different modalities and perform classification. The experiments were conducted on the benchmark DEAP dataset, and the results showed that the classification accuracy of the proposed model in this paper was 95.89% and 94.99% in the arousal dimension and the valence dimension, respectively, which were 2.77% and 3.11% higher compared to the unimodal EEG model, respectively. This paper also analyzed the effects of different peripheral physiological signals on emotion recognition.