Image-Text Alignment and Retrieval Using Light-Weight Transformer

With the increasing demand for multi-media data retrieval in different modalities, cross-modal retrieval algorithms based on deep learning are constantly updated. However, most of them have trouble with large model parameters and insufficient intrinsic nature between different modalities. We proposed a Light-weight Transformer Alignment Network (LTAN), which adopts the current mainstream visual and textual feature extraction methods. With convolutional neural network combined with light-weight transformer architecture and fully connected neural network, LTAN improves the generalization ability of the model while maintaining high performance. In order to extract visual features that lay emphasis on global details, enhancement paths are constructed to fuse precise location signals stored in low-level features with semantic information extracted from high-level to improve the model retrieval accuracy. It obtains the state-of-the-art results on image and sentence retrieval on MS-COCO and Flickr30k datasets. On the MS-COCO 1K test set, our model obtains an improvement of 3.9% and 2.5% respectively for the image and sentence retrieval tasks on the Recall@1 metric. The size of our model is 15% smaller than models using standard transformer as backbone.