Fusion of 3D LIDAR and Camera Data for Object Detection in Autonomous Vehicle Applications

It is vital that autonomous vehicles acquire accurate and real-time information about objects in their vicinity, which fully guarantees the safety of the passengers and vehicle in various environments. Three-dimensional light detection and ranging (3D LIDAR) sensors can directly obtain the position and geometric structure of an object within its detection range, whereas the use of vision cameras is most suitable for object recognition. Accordingly, in this paper, we present a novel object detection and identification method that fuses the complementary information obtained by two types of sensors. First, we utilise 3D LIDAR data to generate accurate object-region proposals. Then, these candidates are mapped onto the image space from which regions of interest (ROI) of the proposals are selected and input to a convolutional neural network (CNN) for further object recognition. To precisely identify the sizes of all the objects, we combine the features of the last three layers of the CNN to extract multi-scale features from the ROIs. The evaluation results obtained on the KITTI dataset demonstrate that: (1) unlike sliding windows that produce thousands of candidate object-region proposals, 3D LIDAR provides an average of 86 real candidates per frame and the minimal recall rate is better than 95%, which greatly decreases the extraction time; (2) The average processing time for each frame of the proposed method is only 66.79 ms, which meets the real-time demand of autonomous vehicles; (3) The average identification accuracies of our method for cars and pedestrians at a moderate level of difficulty are 89.04% and 78.18%, respectively, which is better than those of most previous methods.