Multimodal Fusion of Physiological Signals and Facial Action Units for Pain Recognition

In this paper, we propose a method for pain recognition by fusing physiological signals (heart rate, respiration, blood pressure, and electrodermal activity) and facial action units. We provide experimental validation that the fusion of these signals results in a positive impact to the accuracy of pain recognition, compared to using only one modality (i.e. physiological or action units). These experiments are conducted on subjects from the BP4D+ multimodal emotion corpus, and include same- and cross-gender experiments. We also investigate the correlation between the two modalities to gain further insight into applications of pain recognition. Results suggest the need for larger and more varied datasets that include physiological signals and action units that have been coded for all facial frames.