HearMe: Accurate and Real-time Lip Reading based on Commercial RFID Devices

计算机科学 稳健性(进化) 词汇 频道(广播) 语音识别 延迟(音频) 标识符 人工智能 集合(抽象数据类型) 人机交互 计算机网络 基因 哲学 电信 生物化学 语言学 化学 程序设计语言
作者
Shigeng Zhang,Zijing Ma,Kaixuan Lu,Xuan Liu,Jia Liu,Song Guo,Albert Y. Zomaya,Jian Zhang,Jianxin Wang
出处
期刊:IEEE Transactions on Mobile Computing [IEEE Computer Society]
卷期号:: 1-14 被引量:3
标识
DOI:10.1109/tmc.2022.3208019
摘要

Lip reading can help people with speech disorders to communicate with others and provide them with a new channel to interact with the world. In this paper, we design and implement HearMe , an accurate and real-time lip-reading system built on commercial RFID devices. HearMe can be used to accurately recognize different words in a pre-defined vocabulary without limitations in light conditions and can be used in multiple user scenarios by leveraging RFID's ability in identifying different users. We design an effective data collection strategy to well capture the tiny and complex signal patterns caused by mouth motion and propose a set of algorithms to extract signal profiles related to mouth motions and mitigate interference factors like multi-path. A carefully designed set of features, including time-domain statistical features and frequency-domain features, are then extracted from the signal to lift the recognition accuracy at the word level. To reduce training costs when the model is used in a new environment, a transfer-learning-based approach is adopted to enhance the robustness of the model in cross-environment scenarios. Experimental results show that HearMe detects speaking actions of the user with an accuracy higher than 0.95 and recognizes different words in a 20-words vocabulary with an average accuracy higher than 0.88. Moreover, the latency of HearMe ( $\sim$ 150ms) is nearly two orders of magnitude less than traditional approaches, making it applicable to practical scenarios that require real-time lip reading.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
刚刚
1秒前
麦穗大旋风完成签到,获得积分10
1秒前
星辰大海的应助被zyr采纳,获得10
2秒前
在水一方的应助被科研通管家采纳,获得10
2秒前
今后的应助被科研通管家采纳,获得10
2秒前
在水一方的应助被科研通管家采纳,获得10
3秒前
庞扬完成签到,获得积分20
3秒前
小蘑菇的应助被科研通管家采纳,获得10
3秒前
研友_VZG7GZ的应助被科研通管家采纳,获得10
3秒前
ding的应助被科研通管家采纳,获得10
3秒前
dapang_han发布了新的文献求助10
3秒前
3秒前
云墨完成签到,获得积分20
3秒前
ding的应助被科研通管家采纳,获得10
3秒前
link发布了新的文献求助10
3秒前
3秒前
思源的应助被科研通管家采纳,获得10
3秒前
Xiaohuyan发布了新的文献求助10
4秒前
天天快乐的应助被科研通管家采纳,获得10
4秒前
英俊的铭的应助被科研通管家采纳,获得10
4秒前
慕青的应助被科研通管家采纳,获得10
4秒前
4秒前
科目三的应助被科研通管家采纳,获得10
4秒前
4秒前
领导范儿的应助被科研通管家采纳,获得10
4秒前
4秒前
上官若男的应助被科研通管家采纳,获得10
5秒前
5秒前
koutianle发布了新的文献求助10
5秒前
minya完成签到,获得积分10
5秒前
6秒前
6秒前
壮观问晴完成签到,获得积分10
7秒前
微笑面包完成签到,获得积分10
7秒前
怕孤独的迎波完成签到,获得积分20
8秒前
9秒前
lier发布了新的文献求助10
9秒前
9秒前
10秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
What Else Can I Tell You?: A Pragmatic Study of English Rhetorical Questions as Discursive and Argumentative Acts 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8165074
求助须知:如何正确求助?哪些是违规求助? 9558738
关于积分的说明 21747291
捐赠科研通 7755624
什么是DOI,文献DOI怎么找? 3428780
关于科研通互助平台的介绍 2553162
邀请新用户注册赠送积分活动 2473361