Dynamic Process Planning using Digital Twins and Reinforcement Learning

In order to enable changeable production of Industry 4.0 applications, a production system should respond to unpredictable changes quickly and adequately. This requires process planning to be performed based on the real time operating conditions and dynamic changes to be handled with cognitive skills. To meet this demand, we present a process planning approach using digital twins and reinforcement learning to derive near-optimal process plans. The digital twins enable access to real-time information about the production system. They also constitute the environment for training the agent of the reinforcement learning method. The environment works as a virtual plant, containing the attributes of the product and resources, and uses simulation models of the resources to calculate the reward for an action in terms of reinforcement learning. Reinforcement learning enables our approach to derive process plans via trial and error. Besides the virtual plant, our approach has a planner, which plays the role of the agent to derive near-optimal plans by trying different actions in the virtual plant, and observes the rewards. We apply the Q-learning algorithm to derive near optimal process plans. The evaluation results show that our approach is able to derive near-optimal process plans for different problem sizes. The evaluation also demonstrated the planner's ability to identify by itself which action to take in which situation. Consequently, no modeling of the preconditions and effects of the actions is necessary.