Medical image segmentation based on multi-modal convolutional neural network: Study on image fusion schemes

Motivated by the recent success in applying deep learning for natural image analysis, we designed an image segmentation system based on deep Convolutional Neural Network (CNN) to detect the presence of soft tissue sarcoma from multi-modality medical images, including Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET). Multi-modality imaging analysis using deep learning has been increasingly applied in the field of biomedical imaging and brought unique value to medical applications. However, it is still challenging to perform the multi-modal analysis owing to a major difficulty that is how to fuse the information derived from different modalities. There exist varies of possible schemes which are application-dependent and lack of a unified framework to guide their designs. Aiming at lesion segmentation with multi-modality images, we innovatively propose a conceptual image fusion architecture for supervised biomedical image analysis. The architecture has been optimized by testing different fusion schemes within the CNN structure, including fusing at the feature learning level, fusing at the classifier level, and the fusing at the decision-making level. It is found from the results that while all the fusion schemes outperform the single-modality schemes, fusing at the feature level can generally achieve the best performance in terms of both accuracy and computational cost, but can also suffer from the decreased robustness due to the presence of large errors in one or more image modalities.