Scanning electron microscopy (SEM) images are used to evaluate the microstructure of the concrete, there still remains challenges as the current methods are semi-automated, non-adaptable, and time-consuming, applicable to specific concrete specimens at particular magnification. Therefore, this paper proposes a U-Net convolutional neural network-based automated concrete microstructure analysis to process concrete SEM images for microstructure characterization. An experiment was performed using the SEM images from seven concrete specimens at different magnification to determine the feasibility and evaluate the proposed system performance. The study concluded that the proposed model could segment four concrete components (aggregates, hydrated cement, anhydrous cement, and pores) at an accuracy of 94.43%. Moreover, the average difference of 1.6% and 4.15% was observed between actual and measured degrees of hydration and porosity. The experimental study shows the feasibility of using SEM images and deep learning techniques to automate microstructure analysis quickly, with minimal effort and low-cost. • Concrete microstructure analysis using SEM images and semantic segmentation. • Model identifies aggregates, anhydrous cement, hydrated cement, and pores. • Feasibility validated using SEM images from seven concrete specimens. • Model performance on the testing data is 94.43%. • Method is automatic, accurate, inexpensive, and applicable for different SEM images.