An end-to-end multimodal 3D CNN framework with multi-level features for the prediction of mild cognitive impairment

In recent years, deep learning methods based on brain image have been used for the diagnosis of cognitive impairment-related disorders. With the development of neuroimaging techniques, multi-modality image such as structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) reflect structural and functional information of the brain respectively, and provide more techniques for the diagnosis of cognitive impairment diseases. Combining these complementary image features can lead to more accurate diagnostic assessments compared to using a single modality. Therefore, how to effectively combine multi-modality image features to realize the diagnosis of cognitive impairment disease needs to be further explored. In this work, we propose an end-to-end multimodal 3D CNN framework based on ResNet architecture, which integrates multi-level features obtained under the role of attention mechanisms to better capture subtle differences among brain images, and achieves remarkable diagnostic performance through spatial pyramid pooling strategy and effective fusion of multi-modality features. In this process, we demonstrated that a multimodal framework is more effective by means of non-shared parameters for multi-modality features learning. Moreover, the visualized attention maps show that our model can focus on important brain regions relevant to disease diagnosis. The experimental results demonstrated that our method improved the diagnostic performance in AD diagnosis and MCI conversion prediction by 6.37% and 3.51% compared to the single modality, and it also outperforms some recent state-of-the-art multimodal methods. Especially in AD diagnosis achieved an average accuracy of 94.61%, which provides a more feasible technology for diagnostic assessment of patients with AD.