This study constructs a reinforcement learning-based hybrid algorithm for Aquila Optimizer (AO) and improved Arithmetic Optimization Algorithm (IAOA). The point of the hybrid algorithm is that Q-learning can dynamically select the AO and the IAOA at different stages for different problems. In Arithmetic Optimization Algorithm (AOA), the mathematical optimization acceleration (MOA) function is restructured to balance global search and local exploitation, which can effectively stay away from the local optimum. Moreover, an improved reward function is modeled for Q-learning, which makes our hybrid algorithm more efficient and accurate. A set of benchmark functions and two engineering optimization problems are employed to test the performance of the proposed hybrid algorithm in this paper. Compared with other algorithms, the results show that the proposed hybrid algorithm has higher convergence speed and accuracy.