HearMe: Accurate and Real-time Lip Reading based on Commercial RFID Devices

Lip reading can help people with speech disorders to communicate with others and provide them with a new channel to interact with the world. In this paper, we design and implement HearMe , an accurate and real-time lip-reading system built on commercial RFID devices. HearMe can be used to accurately recognize different words in a pre-defined vocabulary without limitations in light conditions and can be used in multiple user scenarios by leveraging RFID's ability in identifying different users. We design an effective data collection strategy to well capture the tiny and complex signal patterns caused by mouth motion and propose a set of algorithms to extract signal profiles related to mouth motions and mitigate interference factors like multi-path. A carefully designed set of features, including time-domain statistical features and frequency-domain features, are then extracted from the signal to lift the recognition accuracy at the word level. To reduce training costs when the model is used in a new environment, a transfer-learning-based approach is adopted to enhance the robustness of the model in cross-environment scenarios. Experimental results show that HearMe detects speaking actions of the user with an accuracy higher than 0.95 and recognizes different words in a 20-words vocabulary with an average accuracy higher than 0.88. Moreover, the latency of HearMe ( $\sim$ 150ms) is nearly two orders of magnitude less than traditional approaches, making it applicable to practical scenarios that require real-time lip reading.