Spark-based parallel dynamic programming and particle swarm optimization via cloud computing for a large-scale reservoir system

Abstract The joint optimal operation of a large-scale reservoir system is a complex optimization problem with high-dimensional, multi-stage, and nonlinear features. As the number of reservoirs and discrete states increase, the runtime of optimal operation model increases exponentially, leading to the phenomenon of “curse of dimensionality”. Traditional multi-core parallel computing can improve the efficiency to a certain extent, but it is difficult to expand and break through the hardware limitation, which is not suitable for the optimization of the large-scale reservoir system and its refined management. Different from the current literature about reservoir operations that focus on the comparisons of dynamic programming (DP) with particle swarm optimization (PSO) algorithm in serial mode, this paper pays emphasis on a comparison study of parallel DP with parallel PSO via cloud computing. This study proposes the spark-based parallel dynamic programming (SPDP) and spark-based parallel particle swarm optimization (SPPSO) methods via cloud computing. Taking the cascade eight-reservoir system in the Yuanshui basin in China as an example, simulation experiments are carried out for the comparison between SPDP and SPPSO in terms of parallel performance, precision, efficiency, and stability. The results are as follows: (1) The parallel performance of SPDP in the cloud environment is better than SPPSO. (2) Under the same runtime, the precision of SPDP is generally higher than that of SPPSO. (3) Setting the same precision, the runtime of SPPSO is on average 255.18% longer than SPDP, and it does not reach the precision of SPDP. (4) SPPSO has a fast convergence speed and the ability to jump out of the local optimal solution, but its precision increases by 0.41%, while the runtime increases by 229.55% with the increase of iterations. DP solves more accurately and efficiently than PSO via parallel cloud computing, which ensures the global search capability of the algorithm. Moreover, cloud computing is flexible, economical, and safe, with high practical value and application prospects.