In modern manufacturing, quality inspection of object surfaces has already become indispensable in the production. Faster Region Convolutional Neural Network (Faster R-CNN), a deep learning object detection algorithm, has been gradually applied in the field of inspection, but it is of low accuracy in steel surface defect detection. In this paper, a detection method based on improved Faster R-CNN is proposed. In the method, a modified backbone network extracts features from images, deformable convolution kernels replace conventional convolution kernels to make location more precise, and the multi-scale feature layer extracts feature maps of defects in different scales. In the experiment, the solution comes to a mean Average Precision (mAP) of 0.774 on NEU-DET dataset, exceeding the original Faster RCNN model of 0.7 substantially.