CMMix: Cross-Modal Mix Augmentation Between Images and Texts for Visual Grounding

Visual grounding (VG) is a representative multi-modal task that has recently gained increasing attention. Nevertheless, existing works still face challenges leading to under-performance due to insufficient training data. To address this, some researchers have attempted to generate new samples by integrating each two (image, text) pairs, inspired by the success of uni-modal CutMix series data augmentation. However, these methods mix images and texts separately and neglect their contextual correspondence. To overcome this limitation, we propose a novel data augmentation method for visual grounding task, called Cross-Modal Mix (CMMix). Our approach employs a fine-grained mix paradigm, where sentence-structure analysis is used to locate the central noun parts in texts, and their corresponding image patches are drafted through noun-specific bounding boxes in VG. In this way, CMMix maintains matching correspondence during mix operation, thereby retaining the coherent relationship between images and texts and resulting in richer and more meaningful mixed samples. Furthermore, we employ a filtering-sample-by-loss strategy to enhance the effectiveness of our method. Through experiments on four VG benchmarks: ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, the superiority of our method is fully verified.