MLRS-RL: An Energy-Efficient Multilevel Routing Strategy Based on Reinforcement Learning in Multimodal UWSNs

In recent years, multimodal underwater wireless sensor networks (M-UWSNs) have attracted widespread concern in academia. Due to the complex underwater communication environment and the increasing marine applications, it is a crucial issue for M-UWSNs to design an energy-efficient routing strategy that can satisfy multiple transmission latency requirements of different marine applications. Reinforcement learning (RL) approaches with distributed dynamic optimization ability provide a prospective way to solve the aforementioned problem. Therefore, we propose an improved RL framework and then design an energy-efficient multilevel routing strategy (MLRS-RL) based on this framework for multiple transmission latency requirements. In MLRS-RL, a method of model knowledge collection based on the time backoff principle is proposed to preliminarily learn the network environment information before network operates. The convergence speed of the RL framework can be accelerated by utilizing the model knowledge. Then, underwater nodes use the improved RL model to calculate the transmission rewards that data packets with different transmission latency requirements are sent to different candidate relay nodes. Finally, a cooperative transmission strategy using multiple relay nodes is designed to further improve the reliability of data transmission. We verify the effectiveness of the MLRS-RL strategy in terms of packet delivery ratio, transmission latency, energy efficiency, network lifetime, and delivery quantity.