Multi-vessel target tracking with camera fusion for unmanned surface vehicles

With the decreasing availability of sailors, there has been an increasing focus on the development of autonomous ships. Among the various components of autonomous ships, automatic recognition systems that can replace human vision are a crucial area of research. While ongoing studies utilize traditional perception sensors such as RADAR (RAdio Detection And Ranging) and AIS (Automatic Identification System), they have limitations such as blind spots and a restricted detection range. To address these limitations, this paper proposes a new recognition method that utilizes multiple cameras, including electro-optical and infrared radiation cameras, to supplement traditional perception sensors. This method aims to detect maritime obstacles accurately and estimate their dynamic motion using a tracking process. Initially, real-sea images were collected for maritime obstacle detection, and a deep-learning-based detection model was trained on them. The detection results were then employed in an adaptive tracking filter, which allowed the precise motion estimation of the obstacles. Furthermore, to compensate for the limitations of using individual cameras as sensors, this study introduces the simultaneous fusion of tracked data from multiple cameras. This fusion process enhances tracking results in various ways. In field tests using multiple Unmanned Surface Vehicles (USVs), the proposed method successfully converged tracking results within the range of GPS errors. In addition, the fusion of tracked data from multiple cameras significantly improved the tracking results obtained from a single camera.