Reinforcement learning aided geometric shaping and self-canceling coherent detection for a PAM4 FSO communication system

This paper focuses on the tolerability of free space optical (FSO) communication with 4-level pulse amplitude modulation (PAM4) against atmospheric turbulence. Aiming at the trade-off between transmission performance and structral complexity, simplified coherent detection and reinforcement learning aided geometric shaping are proposed to enhance the receiver and transmitter of the PAM4 FSO system, respectively. In the proposed coherent detection structure, the intermediate frequency signal becomes immune to the turbulence-induced phase noise and frequency offset by passing through an electrical square-law device. Then, we find through theoretical analysis and demonstrate that the statistical property of the optical intensity varies among different amplitudes of the PAM4 signal when it is affected by atmospheric turbulence, which indicates a chance that geometric shaping can reduce the turbulence effect. In the geometric shaping scheme, a reinforcement learning algorithm is proposed to determine the optimal set of PAM4 amplitudes that fits the channel conditions. The results demonstrate that the proposed coherent detection structure outperforms direct detection in the bit error rate (BER) by up to one order of magnitude. Combined with the proposed geometric shaping scheme, the BER performance can be further improved. In particular, when the turbulence strength is in the weak to strong range, geometric shaping can improve the BER performance by two orders of magnitude.