Optical vortex beams with fractional orbital angular momentum (OAM) can greatly enhance the channel capacity in free-space optical communication. However, high precision measurement of fractional OAM modes is always difficult, especially under the influence of atmospheric turbulence (AT). In this work, we identify the high-resolution OAM modes down to 0.01 using an improved residual neural network (ResNet) architecture based convolutional neural network (CNN). Experimentally, using a single cylindrical lens, the light intensity distribution can be readily converted into a diffraction pattern containing significant features trained into a CNN model. For the fractional OAM modes from 5.0 to 5.9 over a long propagation distance of 1500 m, at 0.1 resolution, our model's predicting accuracy is up to 99.07% under strong AT, ${C}_{\mathrm{n}}^{2}=1\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}15}\phantom{\rule{4pt}{0ex}}{\mathrm{m}}^{\ensuremath{-}2/3}$. At 0.01 resolution, the accuracy is as high as 86.98% under intermediate AT, ${C}_{\mathrm{n}}^{2}=1\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}16}\phantom{\rule{4pt}{0ex}}{\mathrm{m}}^{\ensuremath{-}2/3}$, and exceeds 73.78% under strong AT, ${C}_{\mathrm{n}}^{2}=1\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}15}\phantom{\rule{4pt}{0ex}}{\mathrm{m}}^{\ensuremath{-}2/3}$. So, these results may have great implications in free-space optical communication.