Cross-modal distillation with audio–text fusion for fine-grained emotion classification using BERT and Wav2vec 2.0

Fine-grained emotion classification for mood- and emotion-related physical-characteristics detection and its application to computer technology using biometric sensors has been extensively researched in the field of affective computing. Although text modality has achieved a considerably high performance from the perspective of sentiment analysis, which simply classifies a positive or negative label, fine-grained emotion classification requires additional information besides text. An audio feature can be adopted as the additional information as it is closely associated with text, and the characteristics of the changes in sound pulses can be employed in fine-grained emotion classification. However, the multimodal datasets related to fine-grained emotion are limited, and the scalability and efficiency are insufficient for multimodal training to be applied extensively via the self-supervised learning (Self-SL) approach, which can adequately represent modality. To address these limitations, we propose cross-modal distillation (CMD), which induces the feature spaces of student models with a few parameters while receiving those of the teacher models that can adequately express each modality based on Self-SL. The proposed CMD performs the mapping of a feature space between teacher-student models based on contrastive learning, while two attention mechanisms—cross-attention between audio and text features and self-attention for features in modality—are performed during knowledge distillation. Wav2vec 2.0 and BERT, which are already adequately trained for audio and text via Self-SL, were adopted as teacher models; audio–text transformer models were used as student models. Accordingly, the CMD-based representation learning applies a lightweight model for IEMOCAP, MELD, and CMU–MOSEI datasets with the task of multi-class emotion classification, while exhibiting better fine-grained emotion classification performance than benchmark models with a considerably low uncertainty for prediction.