A novel apple fruit detection and counting methodology based on deep learning and trunk tracking in modern orchard

Accurate count of fruits is important for producers to make adequate decisions in production management. Although some algorithms based on machine vision have been developed to count fruits which were all implemented by tracking fruits themselves, those algorithms often make mismatches or even lose targets during the tracking process due to the large number of highly similar fruits in appearance. This study aims to develop an automated video processing method for improving the counting accuracy of apple fruits in orchard environment with modern vertical fruiting-wall architecture. As the trunk is normally larger than fruits and appears clearly in the video, the trunk is thus selected as a single-object tracking target to reach a higher accuracy and higher speed tracking than the commonly used method of fruit-based multi-object tracking. This method was trained using a YOLOv4-tiny network integrated with a CSR-DCF (channel spatial reliability-discriminative correlation filter) algorithm. Reference displacement between consecutive frames was calculated according to the frame motion trajectory for predicting possible fruit locations in terms of previously detected positions. The minimum Euclidean distance of detected fruit position and the predicted fruit position was calculated to match the same fruits between consecutive video frames. Finally, a unique ID was assigned to each fruit for counting. Results showed that mean average precision of 99.35% for fruit and trunk detection was achieved in this study, which could provide a good basis for fruit accurate counting. A counting accuracy of 91.49% and a correlation coefficient R2 of 0.9875 with counting performed by manual counting were reached in orchard videos. Besides, proposed counting method can be implemented on CPU at 2 ∼ 5 frames per second (fps). These promising results demonstrate the potential of this method to provide yield data for apple fruits or even other types of fruits.