A Calibration and Real-Time Object Matching Method for Heterogeneous Multi-Camera System

The heterogeneous multi-camera system composed of thermal-infrared camera and visible spectrum (VS) camera can better perceive the surrounding information in complex environment, so it is widely used in many intelligent unmanned devices, such as unmanned aerial vehicles (UAV) and patrol robots. In this paper, a stereo calibration strategy is proposed for this vision system, and a real-time target matching method for heterogeneous images is proposed based on double epipolar constraint. First, a new calibration board is designed, and a calibration points detection and sorting algorithm is proposed. Secondly, the thermal-infrared camera and VS camera are calibrated. Then, the relative position and posture between the two cameras and the projection relationship between the heterologous images are obtained by stereo calibration using the adjusted images, and applied to stereo vision. Finally, the epipolar constraint is applied to the target matching under the heterogenous image combined with the existing target detection technology. In experiments, the average relative error of calibration point positioning using heterogeneous vision system is 0.0209, which is only 0.0014 different from that using homogeneous vision system. When the target appears in the image as a whole and is detected, target matching in a pair of heterologous images only takes 0.068s and the accuracy is 96%, which meeting the real-time and accuracy requirements of general application environments.