Defect detection plays a crucial role in ensuring the surface quality of steel, as it impacts both downstream production and the overall quality of the final product. However, recognizing surface defects in steel has always been challenging due to the small occurrence area, diverse types of deformations, and various types of defects. In this article, we propose a novel lightweight defect detection model named YOLOv5s-DNF, which achieves high detection performance while maintaining a lightweight architecture. Specifically, we enhance the C3 structure in YOLOv5 by incorporating the Deformable Convolution Network v3 (DCNv3) operator, referred to as C3-Dcnv3. This modification effectively addresses the limitations of fixed rectangle structure sampling, enhancing the network's ability to model object deformations. Furthermore, we introduce the Normalized Wasserstein Distance (NWD) to the loss function, improving the model's feature representation and regularization, thereby mitigating overfitting and enhancing generalization ability. In addition, we propose the C3-Faster structure by integrating partial convolution (PConv) into the C3 structure, which reduces the model's parameter count and computational complexity while minimizing precision loss. The proposed YOLOv5s-DNF model is evaluated on the NEU-DET dataset, and the experimental results demonstrate that it achieves a desirable balance between accuracy and parameter efficiency, outperforming other state-of-the-art methods. Furthermore, we conduct additional experiments on the publicly available Wood Surface Defects dataset, further validating the effectiveness of our proposed model. Various experiments have shown that YOLOv5s-DNF achieves a better trade-off between running speed and detection accuracy.