计算机科学
人工智能
模式
多模式学习
模态(人机交互)
机器学习
特征(语言学)
水准点(测量)
判别式
语音识别
任务(项目管理)
代表(政治)
特征学习
模式识别(心理学)
经济
大地测量学
管理
法学
地理
社会学
哲学
政治
语言学
社会科学
政治学
作者
Xuejian Huang,Tinghuai Ma,Li Jia,Yuanjian Zhang,Huan Rong,Najla Al-Nabhan
出处
期刊:Neurocomputing
[Elsevier]
日期:2023-06-06
卷期号:548: 126373-126373
被引量:12
标识
DOI:10.1016/j.neucom.2023.126373
摘要
Intent recognition is a crucial task in natural language understanding. Current research mainly focuses on task-specific unimodal intent recognition. However, in real-world scenes, human intentions are complex and need to be judged by integrating information such as speech, tone, expression, and action. Therefore, this paper proposes an effective multimodal representation and fusion method (EMRFM) for intent recognition in real-world multimodal scenes. First, text, audio, and vision features are extracted based on pre-trained BERT, Wav2vec 2.0, and Faster R-CNN. Then, considering the complementarity and consistency among the modalities, the modality-shared and modality-specific encoders are constructed to learn shared and specific feature representations of the modalities. Finally, an adaptive multimodal fusion method based on an attention-based gated neural network is designed to eliminate noise features. Comprehensive experiments are conducted on the multimodal intent recognition MIntRec benchmark dataset. Our proposed model achieves higher accuracy, precision, recall, and F1-score than state-of-the-art multimodal learning methods. We also conduct multimodal sentiment recognition experiments on the CMU-MOSI dataset, and our model still outperforms state-of-the-art methods. In addition, the experiment demonstrates that the model's multimodal representation well learned the modality's shared and specific features. The multimodal fusion of the model achieves adaptive fusion and effectively reduces possible noise interference.
科研通智能强力驱动
Strongly Powered by AbleSci AI