Residual Deformable Convolution for better image de-weathering

Adverse weather conditions pose great challenges to computer vision tasks like detection, segmentation, tracing et, al. in the wild. Image de-weathering aiming at removing weather degradations from images/videos has hence accumulated huge popularity as a significant component of image restoration. A large number of SOTA de-weathering methods are based on the autoencoder architecture for its excellent generalization and high computational efficiency. However, for most of these models, parts of high-frequency information are inevitably lost in the downsampling process in the encoders, while degraded features are unable to be effectively inhibited in the upsampling modules in the decoders, largely limiting the restoration performance. In this paper, we propose a multi-patch skip-forward structure for the encoder to deliver fine-grain features from shallow layers to deep layers, and provide more detailed semantics for feature embedding. For the decoding part, the Residual Deformable Convolutional module is developed to dynamically recover the degradation with spatial attention, achieving high-quality pixel-wise reconstruction. Extensive experiments show that our model outperforms many recently proposed state-of-the-art works on both specific-task de-weathering, such as de-raining, de-snowing, and all-task de-weathering. The source code is available at github.com/ZhangAoCanada/DeformDeweatherNet.