Two-Domain Joint Attention Mechanism Based on Sensor Data for Group Activity Recognition

In this article, a two-domain joint attention mechanism based on sensor data (TJAMSD) for group activity recognition (GAR) is proposed. We build two networks in the semantic domain and data domain as the teacher network and student network. In the data domain, a GAR network based on graph convolutional network (GCN) with group relation graph (GRG) is proposed. In this network, in order to reflect the relationship between individuals, the individual action feature correlation and position correlation in a group are calculated to construct two relation graphs. Then, the two relation graphs are fused to obtain the final GRG. Finally, the GRG and the individual action features obtained by a hybrid convolutional neural network (CNN) and bi-directional long short-term memory (BLSTM) network are used as the input of the GCN to infer the group activity. Besides, a semantic-domain network is constructed by the known individual action semantics and the group activity semantics. A joint attention mechanism based on the data-domain network and semantic-domain network is proposed. The attention weights learned in the semantic-domain network are used to guide the learning of attention weights in the data-domain network, which allocates attention to different individuals. In this way, TJAMSD makes the networks pay more attention to the key individual actions in the group and overcome the interference caused by noncritical individual actions in GAR. Experiments are conducted on two constructed datasets, the Garsensors dataset and the UT-Data-gar dataset. Different group cases are considered in the experiments and the experimental results show that in all cases, GCN with GRG can better express the interaction features of groups and improve the recognition performance. Furthermore, the TJAMSD can effectively suppress the interference of noncritical actions to advance the model robustness.