Automatic Optimization of Volterra Equalizer With Deep Reinforcement Learning for Intensity-Modulated Direct-Detection Optical Communications

Volterra nonlinear equalizer (VNLE) is widely investigated for linear and nonlinear distortions compensation in optical communication systems. Despite the powerful equalization ability, the required high computation complexity limits its real-time implementation on the hardware. To simplify the complexity, the structure of the VNLE needs to be well optimized. Traditionally, manual optimization of the VNLE structure is blind, and traversal search is inefficient and may result in multiple structures. Therefore, the optimal VNLE structure with the best equalization performance and the lowest equalization complexity for the same performance are still unclear. In this paper, a novel search method named AutoVolterra is proposed to efficiently find the optimal structure of VNLE by using deep reinforcement learning to learn the functional relationship between structure parameters and optimization objectives. Experimental demonstration of 50-Gb/s PAM-4 signal transmission in a bandwidth-limited intensity modulation and direct detection system is performed to verify the effectiveness of AutoVolterra. Both AutoVolterra and greedy search are applied to Volterra feedforward equalizer, Volterra decision feedback equalizer and Volterra-Pruning. The bit error ratio and equalizer's complexity results suggest that the structure searched by AutoVolterra can optimize the equalization performance, and also significantly reduce the complexity, and improve the pruning quality of VNLE to assist real-time implementation on the hardware. Besides, AutoVolterra can also provide a reliable equalization performance benchmark and complexity benchmark to help us verify the actual capabilities of other algorithms.