Multimodal multi-instance learning for long-term ECG classification

Recently, deep learning-based models have been widely used for electrocardiogram (ECG) classification tasks. Most ECG signals are long-term time series that contain a large number of sample points. However, existing deep learning-based models resize or crop the original long-term ECG signal due to the limitation of input size and hardware, which results in information loss. To address this issue, a multimodal multi-instance learning neural network (MAMIL) is proposed for long-term ECG classification. The proposed MAMIL has three major components. First, the original ECG signal and Gramian Angular Field (GAF) image converted from the ECG signal are utilized as multimodal inputs, which enables the model to learn complementary information between different modalities. Second, multi-instance learning (MIL) is introduced to avoid information loss. Each long-term ECG signal and GAF image are treated as bags, and each heartbeat from a long-term ECG signal and each patch from a GAF image are treated as instances. Convolutional neural networks (CNNs) are utilized to extract instance features from different modalities. Third, a novel attention mechanism-based feature fusion method is proposed to aggregate the instance features from multiple modalities to obtain the bag feature for final classification. Our feature fusion method adopts pooling to obtain positive instances, which can effectively eliminate redundant information and achieve low computational complexity. The proposed MAMIL is evaluated on both intrapatient and interpatient patterns of two commonly used ECG datasets. Experimental results show that our model not only outperforms common deep learning-based models, but also outperforms previous MIL-based models.