Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data

Human exposure to reality is multi-modal, and the brain processes it through multi-sensory stimulation. As such, using multi-source intelligence can potentially improve results motivated by human learning. The key challenge in multi-modal learning is to integrate the modalities through a sensible fusion. We propose mBAM - a novel fusion technique inspired by the bottleneck attention module (BAM) to leverage the knowledge from diverse data modes. We combine this module with a deep multi-modal framework for classifying mental disorders. The joint architecture extracts relevant features from diverse inputs - from brain imagery to genomic variables to classify schizophrenia. The model's prediction accuracy is 95.6% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the scheme provides inherent interpretability that helps identify concepts significant for the neural network's decision and explains the underlying factors of the diseases.