To solve the problem of low accuracy and slow speed in the surface defect detection of steel, an improved YOLOv4 algorithm is proposed. Firstly, the SE attention mechanism is embedded between the trunk and neck of the original YOLOv4 model, which can improve the model's ability to capture global information. Secondly, the ICIoU function is introduced to solve the problem that the CIoU function will degenerate when the aspect ratio of the prediction box and the real box is equal. The experiments are carried out on the NEU-DET dataset, the results show that the average accuracy of the improved YOLOv4 algorithm is 78.63%, which is 6.98% higher than that of the original algorithm; the detection speed is 60fps. The model fully meets the requirements of rapid and accurate detection of the production line.