Vehicle flow detection and tracking are crucial components of intelligent transportation systems. However, traditional methods often struggle with challenges such as the poor detection of small objects and low efficiency when processing large-scale data. To address these issues, this paper proposes a vehicle flow detection and tracking method that integrates an improved YOLOv8n model with the ByteTrack algorithm. In the detection module, we introduce the innovative MSN-YOLO model, which combines the C2f_MLCA module, the Detect_SEAM module, and the NWD loss function to enhance feature fusion and improve cross-scale information processing. These enhancements significantly boost the model’s ability to detect small objects and handle complex backgrounds. In the tracking module, we incorporate the ByteTrack algorithm and train unique vehicle re-identification (Re-ID) features, ensuring robust multi-object tracking in complex environments and improving the stability and accuracy of vehicle flow tracking. The experimental results demonstrate that the proposed method achieves a mean Average Precision (mAP) of 62.8% at IoU = 0.50 and a Multiple Object Tracking Accuracy (MOTA) of 72.16% in real-time tracking. These improvements represent increases of 2.7% and 3.16%, respectively, compared to baseline algorithms. This method provides effective technical support for intelligent traffic management, traffic flow monitoring, and congestion prediction.