Multi-branch spatial-temporal-spectral convolutional neural networks for multi-task motor imagery EEG classification

Motor imagery electroencephalograph (MI-EEG) decoding plays a crucial role in developing motor imagery brain-computer interfaces (MI-BCIs). However, MI-EEG signals exhibit temporal variations and spatial coupling characteristics, necessitating effective feature representation for accurate classification. In this paper, we propose a Multi-Task Multi-Branch spatial-temporal-spectral feature representation model based on Convolutional Neural Network (MT-MBCNN) for MI-EEG classification. Our model encompasses three learning tasks: multi-branch spatial-temporal feature classification, multi-bands spectral feature contrastive learning, and class-prototype learning. These tasks are jointly learned during model training, with the losses of each task weighted and optimized to enhance MI-EEG decoding performance. To mitigate the issue of limited samples, we introduce a novel MI-EEG sample augmentation method to augment the diversity of the training set. Extensive experiments are conducted on three publicly available MI-EEG datasets, achieving outstanding average binary classification accuracies of 89.5%, 81.4%, and 70.13% for each dataset, respectively. Ablation studies demonstrate the necessity and significance of multi-task learning, multi-branch architecture, center-loss-based class-prototype learning, and sample augmentation for MI-EEG decoding using CNN models. Our MT-MBCNN model exhibits exceptional capabilities in spatial-temporal-spectral feature representations for constructing MI-BCIs. The source code of MT-MBCNN model is available at: https://github.com/my94my/MT-MBCNN.