IoT-V2E: An Uncertainty-Aware Cross-Modal Hashing Retrieval Between Infrared-Videos and EEGs for Automated Sleep State Analysis

Estimating and monitoring the sleep states at home using ubiquitous infrared (IR) visual camera sensors is an essential healthcare problem. Currently, the common challenge of using IoT sensors to predict sleep stages is the “semantic gap” between the IoT sensory signals and the medical signals, where fewer correlations between IoT sensory signals and the sleep stage labels are observed. To bridge this gap, we propose a novel systematic and methodological IoT design (IoT-V2E) to retrieve the most similar EEG signal representations in a database given an IR visual query for sleep-related analysis. Specifically, we make the following specific contributions: i) we collect a crossmodal retrieval dataset including the IR sensory signals and the synchronized PSG signals with sleep stage ground-truth annotations; ii) we propose a novel uncertainty-aware hashing retrieval method, presenting superior performances, sufficient interpretability, and high memory efficiency; iii) our method achieves the state-of-the-art sleep stage retrieval results and provides the uncertainty for each query in the inference; iv) most importantly, our system is evaluated to be able to assist the physicians not only in diagnosing sleep-related diseases but also finding the subjects with the most similar sleep patterns. Our project is available at https://github.com/SPIresearch/IoT-V2E.