DFGait: Decomposition Fusion Representation Learning for Multimodal Gait Recognition

Multimodal gait recognition aims to utilize various gait modalities for identity recognition. Previous methods have focused on designing complex fusion techniques. However, the heterogeneity between modalities has negatively impacted recognition tasks due to distributional differences and information redundancy. Inspired by this, we have proposed a novel feature decomposition fusion (DFGait) network, combining silhouette and skeleton data. The network learns modality-shared and modality-specific feature representations for both modalities and introduces inter-modality regularization loss and intra-modality regularization loss to encourage the preservation of common and unique information between modalities, reducing modality gaps and information redundancy. Furthermore, the representations mentioned above are embedded in their own space during learning, making the fusion process challenging. Therefore, we have proposed an adversarial modality alignment learning strategy, guiding the alignment of the two modality features through the confusion of the modality discriminator to achieve maximized modality information interaction. Finally, a separable fusion module is introduced to fuse the features of the two modalities, resulting in a comprehensive gait representation. Experimental results demonstrate that our DFGait achieves state-of-the-art performance on popular gait datasets, with rank-1 accuracy of 50.30% for Gait3D and 61.42% for GREW. The source code can be obtained from https://github.com/BoyeXiong/DFGait .