Off-Line Programming Framework for Sorting Task Based on Human-Demonstration

Sorting tasks are typical applications in the product and industrial domains. When facing new settings, such as different types of objects and their positions, the system has to be reprogrammed by experienced engineers, which decreases production efficiency and increases labour and downtime costs. In this paper, an off-line programming framework for sorting tasks based on human demonstration has been developed, taking advantage of machine vision and programming-by-demonstration concepts. Only one eye-to-hand depth camera has been used to record both the demonstrated trajectory and the corresponding relationship of the sorting task. A feature generalization algorithm has also been proposed to ensure that the system is able to reproduce the sorting task using both the key features of the trajectory and the sorting features. Finally, tracking, object and target matching, and generalization experiments have been conducted using the JAKA Zu7 robot with different types of fruit models and target boxes. The results show that the proposed framework is able to record the sorting trajectory at a frequency of 10 Hz, acquire and maintain the sorting relationship after new demonstration, and automatically reproduce the sorting task in new settings. Note to Practitioners —This paper is motivated by a goal to accomplish intuitive and user-friendly off-line programming for sorting task. The programming framework should be usable by non-experts and be able to complete programming tasks with a few demonstrations when faced with new scenarios, such as different types of objects, different correspondence relationships of the object, and different positions of the object. By observing the human demonstrated sorting tasks, the above-mentioned features can be effectively extracted, so as to complete the convenient teaching. The underlying principle of this paper is to let the robot observe human demonstrated sorting task by a depth camera, tracking the relative trajectory between the object and box as well as recording their matching information. The key features of the trajectory are extracted and further generalized combing the matching information to automatically reproduce the executable path in different settings. The proposed framework allows the users without prior programming knowledge to program the sorting task in new setting by simply conducting the pick and place process with their hands.