MTA-YOLACT: Multitask-aware network on fruit bunch identification for cherry tomato robotic harvesting

Accurate and rapid perception of fruit bunch posture is necessary for the cherry tomato harvesting robot to successfully achieve the bunch’s holding and separating. According to the postural relationship of the fruit bunch, bunch pedicel, and plant’ main-stem, the robotic end-effector’s holding region and approach path could be determined, which were important for successful picking operation. The main goal of this research was to propose a multitask-aware network (MTA-YOLACT), which simultaneously performed region detection on fruit bunch, and region segmentation on pedicel and main-stem. The MTA-YOLACT extended from the pre-trained YOLACT model, included two detector branch networks for detection and instance segment, which shared the same backbone network, and the loss function with weighting coefficients of the two branches was adopted to balance the multi-task learning, according to multi-task’s homoscedastic uncertainty during the model training. Furthermore, in order to cluster the fruit bunch, pedicel and main-stem from the same bunch target, a classification and regression tree (CART) model was built, based on the region’s positional relationship from the MTA-YOLACT output. An image dataset of cherry tomato plants in China greenhouse was built to training and test the model. The results indicated a promising performance of the proposed network, with an F1-score of 95.4% on detecting fruit bunches and the mean Average Precision of 38.7% and 51.9% on the instance segmentation of pedicel and main-stem, which was 1.1% and 3.5% more than original YOLACT. Beyond that, our approach performed a real-time detection and instance segmentation of 13.3 frames per second (FPS). The whole bunch could be identified by the CART model with an average accuracy of 99.83% and the time cost of 9.53 ms. These results demonstrated the research could be a viable support to the harvesting robot’s vision unit development and the end-effector’s motion planning in the future research.