The main difficulties of using the deep learning model to decouple the orbital angular momentum (OAM) in Free-Space Optical (FSO) communication are that the model requires a large number of training data sets and the model’s convergence speed is low. In this paper, transfer learning and depthwise separable convolution are combined to improve the computational speed of the model and to reduce the requirement of training data set size. The recognition accuracies of 4-OAM and 8-OAM based on the measured data with noise are studied respectively and the OAM transmission in atmospheric turbulence are simulated to test the robustness of the model. In addition, the proposed method can be trained on the expanded data set of 38 experimental data collection, and the test classification results can reach 99.5%. Meanwhile, the minimum accuracy of the model in testing data of different transmission distances and turbulence intensities is 81.25%, indicating good robustness. The paper’s work on the OAM pattern detection has great significance.