Triboelectric In-sensor Deep Learning for Self-powered Gesture Recognition toward Multifunctional Rescue Tasks

Gesture recognition is an effective method to voice-off communication in daily life, offering an intuitive and immersive means to interactive terminal. Current commercialized solutions to gesture recognition are exposed to limitations of either susceptibility to environmental conditions or the huge cost on fabrication, energy, and computing power. Here, we propose a deep-learning-enabled smart gesture recognition glove system that can capture and recognize the hand gesture in real time and execute the users' interactive instructions for multifunctional rescue tasks. The self-powered gesture recognition glove system can successfully classify 7 typical gestures with an accuracy as high as 98.57%. As a proof of concept, a rescue vehicle equipped with water pump, dexterous robotic arm, or laser radar is demonstrated to be guided by gestures outside the environment to complete rescue tasks even in environments that are inaccessible to humans. We aim to use the self-powered gesture recognition glove for future intelligent interaction and intelligent manufacturing, especially in the diversified scenarios that require more advanced and dexterous manipulations in the dark or blocked spaces.