Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud

Human sensing and understanding is a key requirement for many intelligent systems, such as smart monitoring, human-computer interaction, and activity analysis, etc. In this paper, we present mmParse, the first human parsing design for dynamic point cloud from commercial millimeter-wave radar devices. mmParse proposes an end-to-end neural network design that addresses the inherent challenges in parsing mmWave point cloud (e.g., sparsity and specular reflection). First, we design a novel multi-task learning approach, in which an auxiliary task can guide the network to understand human structural features. Secondly, we introduce a multi-task feature fusion method that incorporates both intra-task and inter-task attention to aggregate spatio-temporal features of the subject from a global view. Through extensive experiments in both indoor and outdoor environments, we demonstrate that our proposed system is able to achieve ~ 92% accuracy and ~ 84% IoU accuracy. We also show that the predicted semantic labels can increase the performance of two downstream tasks (pose estimation and action recognition) by ~ 18% and ~ 6% respectively.