Apple detection algorithm for robotic harvesting using a RGB-D camera

A fruit-by-fruit harvesting robot has not only to locate and characterize the fruits on a tree, but also to detect the obstacles for a collision free operation. Although many researchers have investigated this topic, recognizing and locating fruits on a tree is still a key challenge in developing such a robotic harvesting system due to the occlusion of the target fruits (by foliage, branches or other fruits) and the non-uniform and unstructured environment. Therefore, this study aimed at the development and validation of a system for the localization of apples and obstacles in the orchard based-on the color and depth images acquired with an RGB-D camera (Red, Green, Blue plus Depth). As an RGB-D camera provides both visual and 3D shape information, it is very suitable for 3D perception, which is particularly relevant for robots operating in unstructured environments. Using both color and depth information, an algorithm to detect apples on tree was developed. To separate the pixels belonging to the red or bicolored apples from the foliate background, the redness of each pixel in the color images was measured. Because of its simplicity and efficiency, the Euclidean clustering algorithm has been applied to the depth point cloud to extract the cluster for each apple. Finally, the RANdom Sample Consensus (RANSAC) algorithm has been used to fit each detected cluster to a pre-defined apple model. By using an iterative estimation method to the partially visible target object data (maximum 50% of the target object is observable from a single view-point), RANSAC provides the location of the center, the orientation and diameter of each detected apple. The accuracy of the elaborated algorithm was then tested on RGB-D images acquired in the apple orchard for which ground truth information had also been recorded. With 100% detection rate for the fully visible apples, 85% detection rate for the partially occluded apples and less than 1 cm localization error in a short run time (less than 1 second for the detection of 20 apples), the developed algorithm was found to be suitable for implementation in a robotic harvesting system. Moreover, thanks to its generic approach, this algorithm can also be applied for detecting pre-defined obstacles e.g. trunks, branches, wires. To minimize the memory consumption and calculation time for collision checking, the obstacles are represented in simple geometries such as spheres and cylinders, and stored in a single 3D map with respect to the global coordinates of the robot. The methodology presented in this paper has been tested on an apple picking robot and it has been observed that the developed algorithm can meet the requirements for the robotic harvesting system. Furthermore, the developed algorithm could also be used for yield estimation and orchard monitoring in a management decision support system.