Efficient and accurate pavement surface crack detection is crucial for analyzing pavement survey data. To achieve this goal, an improved lightweight semantic segmentation model based on BiSeNetv2, utilizing the detail branch, the semantic branch, and the guided aggregation module, is refined for automatic pavement surface crack detection. With the detail branch and the semantic branch, the low-level details and the high-level semantic context of pavement surface crack can be represented. Taking advantage of the guided aggregation module, the low-level and high-level crack features are mutually connected and fused. The gradient-weighted class activation mapping (Grad-CAM) is adopted to visualize the details of the evolution of crack feature extraction, fusion, and representation. Based on the evaluation results, the proposed lightweight model demonstrates its effectiveness and robustness in accurately segmenting pavement surface crack. Maximumly, it is 10.14% higher than the other model on F1 score, indicating its great potential for pavement crack detection.