Three-dimensional reconstruction of guava fruits and branches using instance segmentation and geometry analysis

Abstract In unstructured environments, harvesting robots may collide with disorderly growing branches, thus reducing the success rate of harvesting. This study introduces a fruit and branch detection and three-dimensional (3D) reconstruction method for obstacle avoidance path planning of robots. A new architecture for instance segmentation was developed by replacing the backbone of Mask R-CNN with a tiny network, referred to as “tiny Mask R-CNN”. The tiny Mask R-CNN was trained with a small number of images and used to detect guava fruits and branches. Each detected fruit and branch were converted into a 3D point cloud. It was then hypothesized that guava fruits could be represented by 3D spheres and irregular branches can be approximated by a finite number of 3D cylindrical segments. Based on the proposed hypothesis, a random sample consensus-based sphere fitting method and a principal component analysis-based cylindrical segment fitting method were investigated to reconstruct the fruits and branches from the point clouds. A guava dataset with 304 RGB-D images was collected from the fields and used to validate the developed method. The results showed that the detection F1 score of the tiny Mask R-CNN was 0.518; the F1 score for fruit reconstruction was approximately 0.851 and 0.833 under the 2D- and 3D-fruit metrics, respectively; and the F1 score for branch reconstruction was approximately 0.394 and 0.415 under the 2D- and 3D-branch metrics, respectively. These results confirm that the proposed method can effectively reconstruct the fruits and branches and can, therefore, be used to plan an obstacle avoidance path for harvesting robots.