Operational Safe Control for Reinforcement-Learning-Based Robot Autonomy

Reinforcement learning (RL) has been widely used for robot autonomy because it can adapt to dynamic or unknown environments by automatically learning optimal control policies from the interactions between robots and environments. However, the practical deployment of RL can endanger the safety of both robots and environments because many RL methods must experience failures during the training phase. These failures can be reduced or avoided by assuming knowing prior knowledge about the states and environments in the training phase, but this assumption is easily invalid in practical applications, especially with unknown environments. In addition, restarting a training episode could be difficult in practice because the robot may be stuck in the failures. To solve these problems, we propose an operational safe control framework that can automatically recover from failures and reduce failure risks without any prior knowledge. Our framework consists of three steps: (1) detect failures and revert to safe actions, (2) collect correction samples to learn a potential that provides internal environment information to robots, (3) use the potential to shape a safe reward that biases safe explorations. A maze navigation example is used to demonstrate that our method outperforms the traditional reinforcement learning with significantly less failures.