Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Facial landmark detection and action unit~(AU) recognition are two essential tasks in facial analysis. Previous works rarely consider the relationship between these complementary tasks. In this paper, we introduce a novel multi-task dual learning framework to exploit the relationship between facial landmark detection and AU recognition while simultaneously addressing both tasks. When both tasks share middle-level features, common patterns can be exploited and middle- and high-level features can be used to perform facial landmark detection and AU recognition, respectively. In addition, a dual learning mechanism is designed to convert the predicted landmarks and AUs of the label space to the corresponding facial image of the image space, further exploring the strong correlations between the tasks. By jointly training the proposed method at both the feature and label levels, each task improves the other. Experiments on two benchmark databases demonstrate that the proposed method can leverage dependencies to boost the generalization of both tasks.