Fusion of Mask RCNN and attention mechanism for instance segmentation of apples under complex background

It is important to precisely segment apples in an orchard during the growth period to obtain accurate growth information. However, the complex environmental factors and growth characteristics, such as fluctuating illumination, overlapping and occlusion of apples, the gradual change in the ground colour of apples from green to red, and the similarities between immature apples and background leaves, affect apple segmentation accuracy. The purpose of this study was to develop a precise apple instance segmentation method based on an improved Mask region-based convolutional neural network (Mask RCNN). An existing Mask RCNN model was improved by fusing an attention module into the backbone network to enhance its feature extraction ability. A combination of deformable convolution and the transformer attention with the key content only term was used as the attention module in this study. The experimental results showed that the improved Mask RCNN can accurately segment apples under various conditions, such as apples with shadows and different ground colours, overlapped apples, and apples occluded by branches and leaves. A recall, precision, F1 score, and segmentation mAP of 97.1%, 95.8%, 96.4% and 0.917, respectively, were achieved, and the average run-time on the test set was 0.25 s per image. Our method outperformed the two methods in comparison, indicating that it can accurately segment apples in the growth stage with a near real-time performance. This study lays the foundation for realizing accurate fruit detection and long-term automatic growth monitoring.