An Ensemble Approach to Utterance Level Multimodal Sentiment Analysis

The primary objective of sentiment analysis system is to automatically discover and analyze people's attitude, opinion, or position towards a product, a topic, a person or an entity. A huge amount of multimedia content is being posted on social websites such as YouTube, Flicker, and Twitter on every day. To cope up with such multimedia data, there is a need for state-of-the-art multimodal sentiment analysis framework that can extract information from multimodal data. The purpose of this research work is to improve the accuracy of sentiment prediction by analyzing the textual features along with facial expressions. We examine what people say and their facial expressions when they are saying it. Bag-of-words representation is used to create textual features. Facial expressions and audio features were extracted using open source tools such as OpenFace and OpenSmile respectively. Unimodal, bimodal, trimodal and ensemble approaches were used for classification. Our results demonstrate proposed ensemble approach outperforms other base models.