The particle size analysis of aggregate is important in infrastructure construction, while the accurate image segmentation of aggregate particles is one key prerequisite for the reliability of particle size analysis. In this paper, the aggregate particles image segmentation method based on the improved U-Net is proposed to extract aggregate particles edge and aggregate particles mask segmentation image obtained by UNet. Here, the aggregate particles image is used to correct the particles boundary of the mask image. To capture the direction and position of the aggregate particles edge accurately, both global context attention block (GC) and pyramidal convolution (Pyconv) are applied in U-Net to improve the sensitivity of aggregate particles features. To evaluate the performance of our method, we first build a new aggregate particles image segmentation dataset in complex cases. The experimental results show that the proposed method can greatly increase the segmentation effect of aggregate particles image under complex situations.