Canopy segmentation and wire reconstruction for kiwifruit robotic harvesting

Kiwifruits are commercially grown on sturdy support structures such as T-bars and pergolas. Wires are widely used in modern agriculture as an important material for supporting T-bars. It may lead to damage to kiwifruit harvesting end-effector or robot when accessing fruits occluded by branches or wires. Additional development to segment calyxes, branches, and wires will help to achieve higher-level picking strategies. DeepLabV3+ was adopted to segment the fruit calyx, branch, and wire in this work. A method of discrete wire pixels reconstruction was then developed on Progressive probabilistic Hough transform (PPHT) to help sense distribution of the wire. Lines that didn’t meet the constraints, i.e., angle or distance between the lines, were regarded as noise and eliminated. There were 327 images divided into training (261) and testing (66) sets, where the training set was augmented to 1566 images. The dataset was heavily imbalanced where the pixels of calyx, branch, and wire were much fewer than background pixels. For the imbalanced kiwifruit canopy image segmentation, it was proven that the uniform weights assignation method outperformed the median frequency weights. In terms of backbone, ResNet-101 achieved IoUs (intersection over union) of 0.686, 0.709, and 0.424 for calyx, branch, and wire, respectively, and the highest mIoU (mean IoU) of 0.694. It took about 210.0 ms to process a resolution of 512 × 341 pixels image, which could be acceptable for the kiwifruit harvesting robot. The PPHT achieved an correct detection rate of 92.4%, and was competitive in processing time of 6.4 ms/image. Canopy image segmentation can provide a basis for guiding the harvesting end-effector to pick kiwifruits safely, thus improving the harvesting success rate and reducing on-orchard costs.