Deep‐Learning‐Assisted Noncontact Gesture‐Recognition System for Touchless Human‐Machine Interfaces

Abstract Human‐machine interfaces (HMIs) play important role in the communication between humans and robots. Touchless HMIs with high hand dexterity and hygiene hold great promise in medical applications, especially during the pandemic of coronavirus disease 2019 (COVID‐19) to reduce the spread of virus. However, current touchless HMIs are mainly restricted by limited types of gesture recognition, the requirement of wearing accessories, complex sensing platforms, light conditions, and low recognition accuracy, obstructing their practical applications. Here, an intelligent noncontact gesture‐recognition system is presented through the integration of a triboelectric touchless sensor (TTS) and deep learning technology. Combined with a deep‐learning‐based multilayer perceptron neural network, the TTS can recognize 16 different types of gestures with a high average accuracy of 96.5%. The intelligent noncontact gesture‐recognition system is further applied to control a robot for collecting throat swabs in a noncontact mode. Compared with present touchless HMIs, the proposed system can recognize diverse complex gestures by utilizing charges naturally carried on human fingers without the need of wearing accessories, complicated device structures, adequate light conditions, and achieves high recognition accuracy. This system could provide exciting opportunities to develop a new generation of touchless medical equipment, as well as touchless public facilities, smart robots, virtual reality, metaverse, etc.