Improving Image Captioning with Feature Filtering and Injection

Image captioning represents a challenging multimodal task, requiring the generation of corresponding textual descriptions for complex input images. Existing methods usually leverage object detectors to extract visual features of images, and thus utilize text generators for learning. However, the features extracted by these methods lack focus and tend to ignore the relationship between objects and background information. To solve the aforementioned problems, we exploit both region features and grid features of the image to fully leverage the information encapsulated within the images. Specifically, we first propose an Object Filter Module (OFM) to extract the primary visual objects. Furthermore, we introduce a Global Injection Cross Attention (GICA) to inject the global context of the image into the filtered primary objects. The experimental results substantiate the efficacy of our model. Our model’s effectiveness and immense potential have been demonstrated through extensive experimentation on the widely-used benchmark COCO dataset. It outperforms previous methods on the image captioning task, achieving a CIDEr score of 136.1.