RGCN-Based Multi-Label Facial Expression Recognition by Combining Semantic and Geometric Information

Multi-label facial expression recognition (ML-FER) is a challenging problem in affective computing. Due to the physiological structure of the face and the psychological entanglement of emotions, the basic expression elements in blended expressions are interrelated. In this work, we propose a Se[1]mantic and Geometric Multi-Label Relational Graph Convolutional Network (SGML-RGCN), which improves ML-FER by exploiting the relationships between basic expressions. To be specific, R-GCN layers are applied to learn both positive and negative relationships between basic expression categories. Additionally, we utilize the combination of semantic and geometric information of expressions as the node representation in the graph used for R-GCN. Semantic information is provided by word embed[1]dings, and geometric information derives from basic expression features. Experiments on 3 expression datasets: RAF-ML, RAF-compound and JAFFE prove the effectiveness of our method. In addition, we conduct visualization analysis and show the interpretability of our method.