The fact that the existing traffic sign images are easily affected by external factors, and the traffic signs are generally small targets on the images at different scales, has made it difficult in feature extraction when doing traffic sign detection. To achieve better detection results, a multi-target traffic sign detection method with channel attention and feature fusion network (CAFFNet in short) is proposed. This method effectively learns the correlation between feature channels through a lightweight channel attention network, realizes local cross-channel interaction without dimensionality reduction, and enhances the representation ability of the network. The feature pyramid network is used to achieve feature fusion and generate high-resolution multiscale semantic information. The dilated convolution is utilized to capture the multiscale context information to narrow the difference between features and improve the detection effect of the model. The experimental results show that the proposed method on the two datasets GTSDB and CTSD has achieved superior performance in the evaluation criteria compared with the existing detection algorithms.