对话
情态动词
情绪识别
计算机科学
背景(考古学)
心理学
人工智能
语音识别
沟通
历史
化学
考古
高分子化学
作者
Yijing Dai,Jinxing Li,Yingjian Li,Guangming Lu
标识
DOI:10.1016/j.knosys.2024.111954
摘要
Multi-modal emotion recognition in conversation is challenging because of the difficulty to jointly leverage the information from heterogeneous text, acoustic, and visual modalities. Recent context-aware methods usually design a graph structure to model dependencies of utterances and speakers, or integrate Multi-modal information. However, they typically lack a sufficient extraction of unimodal context, and rarely explore the emotion consensus prototypes among different samples with the same label. For solving these problems, in this paper, we propose a Graph Context extraction and Consensus-aware Learning (GCCL) framework to excavate context-sensitive fusion features and simulate the emotion evocation process during the emotion consensus learning. Specifically, GCCL contains a well-designed graph-based module to capture speaker, temporal and modality dependencies and integrate information from different modalities. Then, we design an emotion consensus learning unit to mine the most typical feature of each category in each modality. A speaker-guided contrastive learning loss is further proposed to guarantee the diversity between different individuals and the semantic consistency between distinct modalities. Moreover, we construct a consensus-aware unit with an attention-based memory mechanism to preserve semantic correlations among different samples on the category-level. Extensive experimental results on two conversational datasets demonstrate that the proposed GCCL outperforms the state-of-art methods. Code is available at https://github.com/gityider/GCCL.
科研通智能强力驱动
Strongly Powered by AbleSci AI