Deep Learning–Based Autonomous Road Condition Assessment Leveraging Inexpensive RGB and Depth Sensors and Heterogeneous Data Fusion: Pothole Detection and Quantification

Poor condition of roads is a major factor for traffic accidents and damage to vehicles. A significant portion of car accidents is attributed to severe three-dimensional (3D) pavement distresses such as potholes, ruttings, and ravelings. Insufficient road condition assessment is responsible for the poor condition of roads. To inspect the condition of the pavement surfaces more frequently and efficiently, an inexpensive data acquisition system was developed that consists of a consumer-grade RGB-D sensor and an edge computing device that can be mounted on vehicles and collect data while driving vehicles. The RGB-D sensor is used for collecting two-dimensional (2D) color images and corresponding 3D depth data, and the lightweight edge computing device is used to control the RGB-D sensor and store the collected data. An RGB-D pavement surface data set is generated. Furthermore, encoder-decoder deep convolutional neural networks (DCNNs) consisting of one or two encoders, and one decoder trained on heterogeneous RGB-D pavement surface data are used for pothole segmentation. Comprehensive experiments using different depth encoding techniques and data fusion methods including data- and feature-level fusion were performed to investigate the efficacy of defect detection using DCNNs. Experimental results demonstrate that the feature-level RGB-D data fusion based on the surface normal encoding of depth data outperform other approaches in terms of segmentation accuracy, where the mean intersection over union (IoU) over 10-fold cross-validation of 0.82 is achieved that shows a 7.7% improvement compared with a network trained only on RGB data. In addition, this study explores the efficacy of indirectly using depth information for pothole detection when depth data are not available. Additionally, the semantic segmentation results were utilized to quantify the severity level of the potholes assisting in maintenance decision-making. The result from these comprehensive experiments using an RGB-D pavement surface data set gathered through the proposed data acquisition system is a stepping stone for opportunistic data collection and processing through crowdsourcing and Internet of Things in future smart cities for effective road assessment. Finally, suggestions about the improvement of the proposed system are discussed.