Real-time optimization using reinforcement learning

This work introduces a novel methodology for real-time optimization (RTO) of process systems using reinforcement learning (RL), where optimal decisions in response to external stimuli become embedded into a neural network. This is in contrast to the conventional RTO methodology, where a process model is solved repeatedly for optimality. This reinforcement learning real-time optimization methodology (RL-RTO) utilizes an actor-critic architecture similar to that being used in dynamic control research. However, the methodology presented here is purely for steady-state optimization, which is a novel feature of this work. This work also presents a novel, hybrid training methodology, where a gradient-based optimization solver is used for the training the value network (or critic) and a meta-heuristic optimization algorithm (particle swarm optimization or PSO) is used for training the policy network (or actor). Using this novel training algorithm, the neural networks representing the RL application can be updated in real-time or by using a batch-online training methodology. This technique allows for a solver to utilize the entire data set and attempt to find a global optimum, rather than by taking smaller, incremental update steps after each new data point is collected. As the process system runs and more data becomes available, the critic and the actor networks can be updated in sequence so that the RL-RTO application continually updates itself and gets closer to approaching true optimality. A process system (a chemical reactor) is used as a demonstration case study and also to compare the performance of RL-RTO to a conventional RTO methodology, which uses a near-perfect first principles model of the system, combined with a nonlinear programming (NLP) optimization technique. Each of these methods is compared to a brute force operational methodology in which the system's product throughput is maximized. The RL-RTO application demonstrates promise, as it improves the reactor's annual profit by 9.6%. By comparison, the first principles plus NLP method improves the profit by 17.2%. These RL-RTO results, while promising, indicate that there is still more development needed for RL-RTO to be a viable competitor to the conventional methods.