How Well Apply Multimodal Mixup and Simple MLPs Backbone to Medical Visual Question Answering?

Although current methods have significantly improved the performance of medical visual question answering (Med-VQA), there are still two aspects worth exploring, namely the simplification of model structure and the effective model training on small-scale data. Different from the previous Med-VQA model, this paper only employs multi-layer perceptrons (MLPs) as the backbone network for feature extraction and modal fusion and designs a Med-VQA model on such basis, which achieves superior performance with a simple backbone network. To enhance model generalization, we design multimodal mixup (M-Mixup) to augment images and questions separately, which effectively alleviates the problem of insufficient training samples in the Med-VQA task. To prevent the destruction of the feature relationship when tokenizing the medical image, we design pooling tokens (PTs), a simple downsampling structure to capture fine-grained visual features without affecting the parameters and FLOPs of the entire model. Experimental results demonstrate that our model achieves state-of-the-art on the SLAKE, and obtains a remarkably competitive performance on the VQA-RAD. The source code and models are available at https://github.com/Alivelei/M-Mixup.