MetaFi++: WiFi-Enabled Transformer-Based Human Pose Estimation for Metaverse Avatar Simulation

In the metaverse, digital avatar plays an important role in representing human beings for various interaction with virtual objects and environments, which puts a high demand on effective pose estimation. Though camera-based solutions yield remarkable performance, they encounter privacy issues and degraded performance caused by varying illumination, especially in the smart home. In this article, we propose a WiFi-based Internet of Things-enabled human pose estimation scheme for metaverse avatar simulation, namely, MetaFi++. Specifically, WPFormer is designed with a shared convolutional module and a Transformer block to map the channel state information of WiFi signals to human pose landmarks, effectively exploring spatial information of human pose through self-attention. It is enforced to learn the annotations from the accurate computer vision model, thus achieving cross-modal supervision. Due to the ubiquitous existence of WiFi and robustness to various illumination conditions, WiFi-based human poses are suitable to instruct the movement of digital avatars in the metaverse, promoting avatar applications in smart homes. The experiments are conducted in the real world, and the results show that the MetaFi++ achieves very high performance with a PCK@50 of 97.30%. Our codes are available in https://github.com/pridy999/metafi_pose_estimation .