Deep learning is an effective way to improve the classification accuracy of coal images for the machine vision-based coal sorting. However, the related research on deep learning-based mineral image classification has not systematically considered the models for multi-coal and multi-class sorting. Additionally, the universal CNNs model for multi-coal image classification has not been proposed. Given the above problems, combined with deep learning and transfer learning and based on VGG Net, Inception Net, and Res Net, this study builds four CNNs models with different depth and structure for multi-coal and multi-class image classification. Finally, we take anthracite, gas coal, coking coal as the research objects and propose a universal CNNs model suitable for multi-coal and multi-class sorting. Moreover, with the Channel Visualization map, Heatmap, Gard-CAM map, and Guided Backpropagation map, the operational processes of CNNs model in coal image recognition and classification are revealed, and the features that affect the classification weights are analyzed.