SimEmotion: A Simple Knowledgeable Prompt Tuning Method for Image Emotion Classification

Image emotion classification is an important computer vision task to extract emotions from images. The state-of-the-art methods for image emotion classification are primarily based on proposing new architectures and fine-tuning them on pre-trained Convolutional Neural Networks. Recently, learning transferable visual models from natural language supervision has shown great success in zero-shot settings due to the easily accessible web-scale training data, i.e., CLIP. In this paper, we present a conceptually simple while empirically powerful framework for supervised image emotion classification, SimEmotion, to effectively leverage the rich image and text semantics entailed in CLIP. Specifically, we propose a prompt-based fine-tuning strategy to learn task-specific representations while preserving knowledge contained in CLIP. As image emotion classification tasks lack text descriptions, sentiment-level concept and entity-level information are introduced to enrich text semantics, forming knowledgeable prompts and avoiding considerable bias introduced by fixed designed prompts, further improving the model’s ability to distinguish emotion categories. Evaluations on four widely-used affective datasets, namely, Flickr and Instagram (FI), EmotionROI, Twitter I, and Twitter II, demonstrate that the proposed algorithm outperforms the state-of-the-art methods to a large margin (i.e., 5.27% absolute accuracy gain on FI) on image emotion classification tasks.