Co-dance with Ambiguity: An Ambiguity-Aware Facial Expression Recognition Framework for More Robustness

Facial Expression Recognition (FER) has received considerable research attention owing to its poor robustness in real-world scenarios. This issue, defined as the uncertainty problem in FER, is often solved by recognizing the noise samples in FER datasets. Unlike noise samples with incorrect labels, ambiguous samples exhibit mixed emotions that align with multiple basic expressions. It makes them indistinguishable in training and harms model robustness. To address this issue, we propose an ambiguity-aware FER framework called Co-dance with Ambiguity (CoA). CoA combines an Emotion Extraction Module (EEM) and an Expression Description Module (EDM) to leverage ambiguity for better performance and robustness. Specifically, EEM employs a coupled-stream structure to extract both representative and detailed features through diverse-scale fusion and patch-attention sensing. EDM adjusts ground-truth labels of ambiguous samples by introducing label pairs derived from the top two highest predictions, describing the mixed-emotion nature. The pairs guide the model to align feature extraction with the inherent ambiguity of ambiguous samples during training. Extensive experiments on five in-the-wild FER datasets demonstrate the superiority of CoA over advanced methods. Moreover, introducing ambiguity-aware strategies enriches feature representations and significantly enhances robustness when faced with a high ratio of ambiguous samples in FER.