This article introduces Domain-fusion YOLO (DF-YOLO), a novel object detection network enhancing YOLOv5 series networks’ generalizability, particularly in fabric defect detection. DF-YOLO incorporates a unique feature extractor in its backbone, enhancing frequency domain feature perception alongside spatial information. Utilizing Fourier transform, the network better discerns fabric textures and anomalies, addressing traditional convolutional neural networks’ limited receptive field issue. Feature fusion and dimension reduction are applied for capturing latent features. In its neck, DF-YOLO integrates an improved BiFPN-based feature-fusion structure and CBAM attention mechanism, optimizing feature selection and fusion across scales. This mitigates information loss due to varying input resolutions and architecture depth. DF-YOLO marks a significant advancement in textile anomaly detection, improving generalization while maintaining efficiency. Its innovative approach benefits automated quality control, offering a sophisticated tool for industries requiring precise fabric inspection.