Multi-scale driver behavior modeling based on deep spatial-temporal representation for intelligent vehicles

Abstract The mutual understanding between driver and vehicle is critical to the realization of intelligent vehicles and customized interaction interface. In this study, a unified driver behavior modeling system toward multi-scale behavior recognition is proposed to enhance the driver behavior reasoning ability for intelligent vehicles. Specifically, the driver behavior recognition system is designed to simultaneously recognize the driver's physical and mental states based on a deep encoder-decoder framework. The model jointly learns to recognize three driver behaviors with different time scales: mirror checking and facial expression state, and two mental behaviors, including intention and emotion. The encoder module is designed based on a deep convolutional neural network (CNN) to capture spatial information from the input video stream. Then, several decoders for different driver states estimation are proposed with fully-connected (FC) and long short-term memory (LSTM) based recurrent neural networks (RNN). Two naturalistic datasets are used in this study to investigate the model performance, which is a local highway dataset, namely, CranData, and one public dataset from Brain4Cars. Based on the spatial–temporal representation of driver physical behavior, it shows that the observed physical behaviors can be used to model the latent mental behaviors through the proposed end-to-end learning process. The testing results on these two datasets show state-of-the-art results on mirror checking behavior, intention, and emotion recognition. With the proposed system, intelligent vehicles can gain a holistic understanding of the driver's physical and phycological behaviors to better collaborate and interact with the human driver, and the driver behavior reasoning system helps to reduce the conflicts between the human and vehicle automation.