Accurate pavement crack segmentation is crucial for civil engineering and infrastructure maintenance. To address the challenge of imbalanced data resulting from the prevalence of non-crack pixels, this research seeks to improve the quality of pavement crack segmentation, particularly for thick and tiny cracks. This paper presents an Asymmetric Dual-Decoder-U-Net (ADDU-Net) model, which involves constructing an asymmetric dual decoder with a dual attention module to better capture the features of both thick and tiny cracks under diverse environmental conditions. Through evaluation with images from four benchmark datasets, the ADDU-Net model demonstrates its effectiveness and robustness in accurately segmenting various types of cracks. This segmentation model shows significant potential for improving crack segmentation in industrial applications.