With the continuous increase of surface quality requirements in steel industry manufacturing, defect detection has received extensive attention. Correct and rapid detection of steel surface defects can significantly improve product quality and productivity. The existing methods improve accuracy by expanding the depth of networks or using various feature fusion technologies, but reduce the computational efficiency. To achieve the balance of precision and speed, we propose an improved network based on Faster R-CNN for defect detection of steel surface. Firstly, the emerging ConvNeXt architecture is adopted to act as backbone to extract features in Faster R-CNN. Besides, Convolutional Block Attention Module (CBAM) is used to improve the attention of our model to surface defects and suppress features of the complex background. Finally, k-means clustering algorithm is utilized to generate anchors that are better adapted to the surface defects. The proposed method achieves a mean average precision (mAP) of 80.78% and a detection speed of 26 frames per second (FPS) on the NEU-DET dataset, improving about 1.5% compared to the YOLOv5 and 8.4 % compared to original Faster R-CNN, which indicates its superior for defect detection of steel surface.