Language Guided Graph Transformer for Skeleton Action Recognition

The Transformer model is a novel neural network architecture based on a self-attention mechanism, primarily used in the field of natural language processing and is currently being introduced to the computer vision domain. However, the Transformer model has not been widely applied in the task of human action recognition. Action recognition is typically described as a single classification task, and the existing recognition algorithms do not fully leverage the semantic relationships within actions. In this paper, a new method named Language Guided Graph Transformer (LGGT) for Skeleton Action Recognition is proposed. The LGGT method combines textual information and Graph Transformer to incorporate semantic guidance in skeleton-based action recognition. Specifically, it employs Graph Transformer as the encoder for skeleton data to extract feature representations and effectively captures long-distance dependencies between joints. Additionally, LGGT utilizes a large-scale language model as a knowledge engine to generate textual descriptions specific to different actions, capturing the semantic relationships between actions and improving the model’s understanding and accurate recognition and classification of different actions. We extensively evaluate the performance of using the proposed method for action recognition on the Smoking dataset, Kinetics-Skeleton dataset, and NTU RGB $$+$$ D action dataset. The experimental results demonstrate significant performance improvements of our method on these datasets, and the ablation study shows that the introduction of semantic guidance can further enhance the model’s performance.