Fusion of CT images and clinical variables based on deep learning for predicting invasiveness risk of stage I lung adenocarcinoma

To develop a novel multimodal data fusion model by incorporating computed tomography (CT) images and clinical variables based on deep learning for predicting the invasiveness risk of stage I lung adenocarcinoma that manifests as ground-glass nodules (GGNs) and compare the diagnostic performance of it with that of radiologists.A total of 1946 patients with solitary and histopathologically confirmed GGNs with maximum diameter less than 3 cm were retrospectively enrolled. The training dataset containing 1704 GGNs was augmented by resampling, scaling, random cropping, and so forth, to generate new training data. A multimodal data fusion model based on residual learning architecture and two multilayer perceptron with attention mechanism combining CT images with patient general data and serum tumor markers was built. The distance-based confidence scores (DCS) were calculated and compared among multimodal data models with different combinations. An observer study was conducted and the prediction performance of the fusion algorithms was compared with that of the two radiologists by an independent testing dataset with 242 GGNs.Among the whole GGNs, 606 GGNs are confirmed as invasive adenocarcinoma (IA) and 1340 are non-IA. The proposed novel multimodal data fusion model combining CT images, patient general data, and serum tumor markers achieved the highest accuracy (88.5%), area under a ROC curve (0.957), F1 (81.5%), F1weighted (81.9%), and Matthews correlation coefficient (73.2%) for classifying between IA and non-IA GGNs, which was even better than the senior radiologist's performance (accuracy, 86.1%). In addition, the DCSs for multimodal data suggested that CT image had a stronger influence (0.9540) quantitatively than general data (0.6726) or tumor marker (0.6971).This study demonstrated that the feasibility of integrating different types of data including CT images and clinical variables, and the multimodal data fusion model yielded higher performance for distinguishing IA from non-IA GGNs.