Multimodal Emotion Recognition in Response to Oil Paintings

Most previous affective studies use facial expression pictures, music or movie clips as emotional stimuli, which are either too simplified without contexts or too dynamic for emotion annotations. In this work, we evaluate the effectiveness of oil paintings as stimuli. We develop an emotion stimuli dataset with 114 oil paintings selected from subject ratings to evoke three emotional states (i.e., negative, neutral and positive), and acquire both EEG and eye tracking data from 20 subjects while watching the oil paintings. Furthermore, we propose a novel affective model for multimodal emotion recognition by 1) extracting informative features of EEG signals from both the time domain and the frequency domain, 2) exploring topological information embedded in EEG channels with graph neural networks (GNNs), and 3) combining EEG and eye tracking data with a deep autoencoder neural network. From the exper-iments, our model obtains an averaged classification accuracy of 94.72 % ± 1.47 %, which demonstrates the feasibility of using oil paintings as emotion elicitation material.