Micro-expression Recognition Based on Facial Graph Representation Learning and Facial Action Unit Fusion

Micro-expressions recognition is a challenge because it involves subtle variations in facial organs. In this paper, first, we propose a novel pipeline to learn a facial graph (nodes and edges) representation to capture these local subtle variations. We express the micro-expressions with multi-patches based on facial landmarks and then stack these patches into channels while using a depthwise convolution (DConv) to learn the features inside the patches, namely, node learning. Then, the encoder of the transformer (ETran) is utilized to learn the relationships between the nodes, namely, edge learning. Based on node and edge learning, a learned facial graph representation is obtained. Second, because the occurrence of an expression is closely bound to action units, we design an A U-GCN to learn the action unit’s matrix by embedding and GCN. Finally, we propose a fusion model to introduce the action unit’s matrix into the learned facial graph representation. The experiments are comparing with SOTA on various evaluation criteria, including common classifications on CASME II and SAMM datasets, and also conducted following Micro-expression Grand Challenge 2019 protocol.