Joint recognition of basic and compound facial expressions by mining latent soft labels

Previous works on facial expression recognition focus on basic emotions, while ignoring more complex compound expressions. However, both basic and compound emotions appear in the real-world environment. In this work, we aim to jointly recognize basic and compound expressions. Aiming at the Basic-Compound Facial Expression Recognition (BC-FER) task, we illustrate that traditional hard label training is not ideal due to great label dependencies. Therefore, we propose an expression soft label mining (ESLM) method to improve the performance. On the one hand, an iterated soft label mining (ISLM) algorithm assisted by teacher–student network is proposed to make the network generate soft targets automatically for learning. On the other hand, to explicitly leverage prior knowledge of label correlations, we propose an expression correlation score learning (ECSL) loss to regularize the predicted distributions. Extensive experimental results on CFEE, RAF-DB, and EmotioNet show that our method achieves state-of-the-art performance on BC-FER task.