MS3DAAM: Multi-scale 3-D Analytic Attention Module for Convolutional Neural Networks

In this paper, we propose a compact and effective module, called multi-scale 3-D analytic attention module (MS3DAAM) to address this challenge. We significantly reduce model complexity by developing a decoupling-and-coupling strategy. Firstly, we factorize the regular attention along channel, height and width directions and then efficiently encode the information via 1-D convolutions, which greatly saves the computational power. Secondly, we multiply the weighted embedding results of the three direction vectors to regain a better 3-D attention map, which allocates an independent weight to each neuron, thus developing a unified measurement method for attention. Furthermore, multi-scale method is introduced to further strengthen our module capability in locating by capturing both the inter-channel relationships and long-range spatial interactions from different receptive fields. Finally, we develop a structural re-parameterization technique for multi-scale 1-D convolutions to boost the inference speed. Extensive experiments in classification and object detection verify the superiority of our proposed method over other state-of-the-art counterparts. This factorizing-and-combining mechanism with the beauty of brevity can be further extended to simplify similar network structures.