DAformer: Transformer with Domain Adversarial Adaptation for EEG-Based Emotion Recognition with Live-Oil Paintings

The emergence of domain adaptation has brought remarkable advancement to EEG-based emotion recognition by reducing subject variability thus increasing the accuracy of cross-subject tasks. A wide variety of materials have been employed to elicit emotions in experiments, however, artistic works that aim to evoke emotional resonance of observers are relatively less frequently utilized. Previous research has shown promising results in electroencephalogram(EEG)-based emotion recognition on static oil paintings. As video clips are widely recognized as the most commonly used and effective stimuli, we adopted animated live oil paintings, a novel set of emotional stimuli in the live form which are essentially a type of video clip while possessing fewer potential influencing factors for EEG signals compared to traditional video clips, such as abrupt switches on background sound, contrast, and color tones. Moreover, previous studies on static oil paintings focused primarily on the subject-dependent task, and further research involving cross-subject analysis remains to be investigated. In this paper, we proposed a novel DAformer model which combines the advantages of Transformer and adversarial learning. In order to enhance the evocative performance of oil paintings, we introduced a type of innovative emotional stimuli by transforming static oil paintings into animated live forms. We developed a new emotion dataset SEED-LOP (SJTU EEG Emotion Dataset-Live Oil Painting) and constructed DAformer to verify the effectiveness of SEED-LOP. The results demonstrated higher accuracies in three-class emotion recognition tasks when watching live oil paintings, with a subject-dependent accuracy achieving 61.73% and a cross-subject accuracy reaching 54.12%.