Because unmanned aerial vehicles(UAVs) have the characteristics of low flight trajectory, slow motion speed, and small volume, they are difficult to identify using current vision technologies. To meet the requirements for detection speed and accuracy in the field of UAV detection, we propose an improved You Only Look Once version 5(YOLOv5) UAV detection method. It uses visible light and infrared imaging datasets for daytime detection and nighttime detection respectively. Based on these two datasets, our model can improve detection accuracy in challenging environments such as dim light or nighttime. We choose a model pruning method based on the Batch Normalization(BN) layers to prune the YOLOv5 detection model to improve the detection speed of the model. To solve the problem of misjudging birds as UAVs, EfficientNet is added to re-classify the detection results of YOLOv5. We constructed a dataset containing over 10000 visible light images and over 10000 infrared imaging images to evaluate the performance of the proposed algorithm. Qualitative and quantitative experimental results show that the proposed algorithm has greatly improved the accuracy and speed of UAV detection.