Classification of Hateful Memes Using Multimodal Models

Nowadays memes have become popular and are a convenient medium for internet communication. They can spread ideas over the internet and influence people in no time. Most memes are humorous, but some of them may also contain hatred which in some way or the other can be offensive to some people. Hence, an algorithm that identifies offensive memes on the social media platform to avoid spreading hate is required. The classification into hateful or non-hateful memes and analysis of such memes is currently an active domain for researchers. A Meme can be hateful when read along with the image and text combined, which implies that the meme may not be hateful if one pays attention to only the image or text in the meme. Therefore, there is a need for a multimodal approach that understands the relativeness of visual and language information present in the meme. Our work in this paper focuses on the multimodal classification of hate memes using fusion techniques. We have used the dataset provided by Facebook AI for its hateful memes challenge. We use the early fusion technique to combine the image and text modality to build a classifier for this project. For fusion, we have used the baseline models for classification of both image and text that are Inception v3 and BERT, respectively. And we were able to achieve 0.79 as AUC score with 63.3 percent of model accuracy.