An improved obstacle separation method using deep learning for object detection and tracking in a hybrid visual control loop for fruit picking in clusters

Selectively picking a target fruit surrounded by obstacles remains a challenge for fruit harvesting robots. This paper presents improvements to the active obstacle separation method for strawberry picking in clusters. A faster and more accurate vision system was developed that combined two neural networks and color thresholding for real-time detection, tracking and localization of strawberries. We propose an improved active obstacle separation method that used a push and a drag-push operation to separate the obstacles from the target in three stages. The push and drag vectors were simplified and precisely calculated based on the exact locations of obstacles. Also, different from many systems that only “looked” once for the entire picking process, the new system used a hybrid vision-based control method. In stage 1, the push operation was controlled by a simple closed-loop vision at two key points. In stages 2 and 3, the vision system re-perceived the environment to update the target and obstacle information before each round of drag-push movements. Field evaluation showed that the proposed method was more precise to separate the obstacles without reducing the speed, increasing the whole process success rate to 62.4% in clusters on the “Murano” strawberry cultivator that was 36.8% higher than the previous work.