Multimodal feature fusion for illumination-invariant recognition of abnormal human behaviors

This paper presents a fusion of feature maps extracted from visible and thermal infrared surveillance videos to recognize abnormal human behaviors in low illumination conditions. Abnormal human behaviors often occur at night or in darkness, where a visible camera alone may not capture sufficient visual information for analysis. The proposed scheme uses additional data captured by a thermal infrared camera to enhance information on human appearance and motion. Thermal and visible data are assumed to be informative for analyzing human behaviors, yet they contribute differently to certain areas of video frames. This paper proposes a multimodal feature fusion (MFF) to produce a weight matrix that determines how much thermal or visible input should contribute to obtain optimal performance. Experiment results demonstrate that MFF effectively recognizes abnormal human behaviors in darkness. MFF achieves an overall accuracy of 91.01%, which is at least 4.53% higher than individual thermal or visible data and 2.17% higher than the state-of-the-art fusion-based abnormal human behavior recognition techniques.