The detection of defect types of photovoltaic (PV) panel is a crucial task in PV system. Existing detection models face challenges in effectively balancing the trade-off between detection accuracy and resource consumption. To address this issue, this paper proposes a new defect detection method for PV panel based on the improved YOLOv8 model, which realizes both the high detection accuracy and the lightweight. Firstly, Reversible Column Networks (RevCol) is used as the Backbone of YOLOv8, which makes sure to preserve the feature information in the process of network transmission and also reduces the number of parameters and Giga floating-point operations per second (GFLOPs). Subsequently, a new lightweight Bottleneck fused with Efficient Multi-Scale Attention (EMA) is designed to optimize the CSPDarknet53 to 2-Stage FPN (C2f) module of Neck in YOLOv8 to enhance the robustness and further decrease network parameters. Finally, Squeeze-and-Excitation (SE) Attention is integrated into the Head of YOLOv8 to prioritize the important channel features and thus enhance the detection performance. The experimental results on the PVEL-AD dataset demonstrate that parameters and GFLOPs of the proposed model are declined by 38.46% and 34.39% respectively, and mAP0.5:0.95 is increased by 2.6% compared with the baseline model. The lightweight improved YOLOv8 model facilitates the deployment of deep learning model on edge devices and provides a novel approach for the online detection of PV panel defects.