Art Image Inpainting with Style-guided Dual-branch Inpainting Network

Traditionally, art images have to be restored by professionals for a very long time. It is also possible to maintain the artistic value of damaged art images by digitizing them and restoring them through computer-aided means. However, existing advanced image inpainting methods are mainly intended for natural images and are not suitable for art images. Thus, we propose a novel style-guided dual-branch inpainting network (SDI-Net) to address the above-mentioned issue. Specifically, our SDI-Net consists of a style reconstruction (SR) branch and a style inpainting (SI) branch, in which the SR branch provides intermediate supervision (style and content supervision) for the SI branch. The SI branch performs art image inpainting with a coarse-to-fine approach. At the coarse inpainting stage, the content and style of art image are separated and preliminarily inpainted under the supervision of SI branch. In addition, we propose a class style learning (CSL) module to inpaint the style feature guided by the style label, which can provide more effective brushstrokes from the same class of art images. The coarse inpainted results can be obtained by fusing the inpainted style feature with the inpainted content feature. At the fine inpainting stage, a style attention (SA) module is proposed in the decoder to further refine the coarse inpainted results. We employ the style loss, the content loss, the multi-class style adversarial loss, and the reconstruction loss to jointly train the proposed SDI-Net. A variety of experiments demonstrate the effectiveness of the proposed method, which allows the filled brushstrokes to appear as realistic as possible.