This paper mainly studies the semantic segmentation algorithm for smart car scenes. Considering the real-time nature of the semantic segmentation algorithm in smart car scenes, this paper proposes a real-time semantic segmentation model based on an improved lightweight network. On the basis of the lightweight network MobileNet v2, a network with dilated convolution and attention modules is constructed as the backbone network for feature extraction, and pooling operations of different sizes are used on the dilated pyramid pooling module ASPP to encode surrounding Semantic information. On this basis, a lightweight semantic segmentation model LightSeg was constructed. Tested on a single RTX 2080Ti GPU, the model achieves a segmentation accuracy of 73.8% mIoU at a speed of 49.5 frames per second on the Cityscapes dataset.