A Convolutional-Transformer based Approach for Dynamic Gesture Recognition of Data Gloves

Data glove-based dynamic gestures contain rich human motion intentions, which is reliant on the hand body information that comes from multi-individual sensors attached. However, present gesture recognition with such wearable sensor devices tends to depend heavily on the handcrafted features and ignore the critical channel and inter-feature information. To address this problem, a novel convolutional-transformer based recognition architecture termed as the spatial-temporal feature-attention transformer network (STFTnet) is proposed in this study. Specifically, the acquired data from multiple sensors of the data glove are sequentially processed with a spatial-temporal sensor features embedding branch, a transformer encoder block, and the final gesture classifier. A multi-sensor feature attention (MFA) block and an improved depth-separable convolution block of the first branch are developed to effectively extract low-level spatial and local temporal features, while the multi-head self-attention based transformer block further concentrating on capturing the global context information. The gesture classifier is used to achieve the final classification successfully. To evaluate the efficacy of the proposed approach, extensive experiments are conducted on two publicly available datasets of pelvic closed reduction action dataset and UC2017 Hand Gesture Dataset, and one self-built gesture control command dataset. Compared to the other state-of-the-art deep learning-based algorithms, an average accuracy of 95.75%, 100%, 99.72% and recognition time of 10.71ms, 11.92ms, and 11.24ms has been achieved. These results indicate that the proposed network effectively enhances the recognition performance of the dynamic gesture of data gloves, while fulfilling requirements of the further real-time application.