3D Mesh classification and panoramic image segmentation using spherical vector networks with rotation-equivariant self-attention mechanism

Spherical signals exist in many applications such as planetary data, lidar scanning and digitization of 3D objects, so we need models that can effectively process spherical data. When the spherical data is simply projected onto a two-dimensional plane and then convolutional neural networks (CNNs) are used, the performance of the previous algorithms that exist in the literature is poor due to the distortion caused by the projection and the invalid translational equivariance. We propose a spherical vector network with rotation-equivariant self-attention mechanism for part-whole relationships learning to avoid a certain degree of distortion in this paper. Specifically, we take first the spherical convolutional network as the front-end network to obtain primary vectors, then we achieve the part-whole relationships between vectors through proposed rotation-equivariant self-attention mechanism to obtain advanced vectors which can represent the existence probability of the entity and orientations. Experimental results show that the proposed method combined with the front-end network improves the 3D mesh classification accuracy of the front-end network by 9% when the training set is not rotated and the test set is rotated arbitrarily under the rigid ModelNet40 dataset. Similarly, the 3D mesh classification accuracy of the front-end network improves by 12.2% under the non-rigid SHREC15 dataset. In addition, our method is compared with the recent method in the spherical image semantic segmentation task, achieving an improvement of 2.2% in mean pixel accuracy and 1.3% in mean intersection over union.