Towards visual emotion analysis via Multi-Perspective Prompt Learning with Residual-Enhanced Adapter

Visual emotion analysis (VEA), a crucial task that identifies the emotional content conveyed in an image, has garnered increasing attention due to the rapid growth of image posts in modern social media. Prompted by the remarkable generalization ability of prompt learning, recent studies have endeavored to develop adaptive contextual prompts to adjust the representation spaces in visual emotion analysis dynamically. However, the majority of these studies have focused on non-oriented prompting strategies, overlooking the sentiment information associated with emotion labels. In this paper, we propose a novel approach, Multi-Perspective Prompt Learning (MPP-CLIP), within the context of CLIP, for visual emotion analysis. Our approach not only involves non-oriented prompting using learnable contexts but also leverages emotion-oriented prompting based on emotion labels. Thus, our design facilitates a strong coupling between the vision-emotion features, ensuring mutual synergy and discouraging the adoption of solely non-oriented solutions. In addition, we introduce Residual-Enhanced Adapter, which employs residual-style feature mixing to achieve efficient transfer learning from pre-trained CLIP to visual emotion analysis. Extensive experimental evaluations demonstrate our approach outperforms previous state-of-the-art (SOTA) methods while maintaining significantly lower computational and memory costs.