EEG Guided Multimodal Lie Detection with Audio-Visual Cues

Lying is considered a form of deception that defines one of the inevitable parts of human essence. Also, deception or lie detection has numerous applications in criminal and judicial community. Traditional practices of identifying deceit includes the monitoring of physiological signals, transcripts, visual and acoustic information with scientific techniques. In this paper, we propose a multimodal lie detection system that leverage the capabilities of novel deep learning techniques. In particular, the study investigates the importance of visual, acoustic and EEG information of a human subject for deception detection task. On the vision side, the system extracts dense optical flow features from consecutive frames in a video to monitor the facial movements. A two-stream convolution neural network utilize this visual features to detect lie or truth. Speech based deceit identification system extracts frequency distributed spectrograms from audio signals and attention augmented CNN is employed to learn changes in distribution of frequencies in speech. For lie detection with EEG signals, we utilize bidirectional long short term neural network for representation and classification of EEG data. EEG signals are represented as time series data and Bi-directional LSTM is learns the correspondences of past signals and future signals. The study performs multimodal fusion on all modalities for lie detection with best performing classifier. Experiments on Bag-Of-Lies dataset showed that the system outperformed traditional machine learning approaches with a significant difference. When all modalities are combined, the system achieves an accuracy of 83.5% in distinguishing deceptive and truthful samples.