Exemplar-based depth inpainting with arbitrary-shape patches and cross-modal matching

Commodity RGB-D cameras can provide texture and depth maps in real-time, and thus have facilitated the booming development of various depth-dependent applications. However, depth maps suffer from the loss of valid values, which leads to holes and impairs both research and applications. In this paper, we propose a novel exemplar based method to fill depth holes and thus to improve depth quality. This novel method is based on the fact that a depth map has many similar even identical parts, and the lost depth values can be restored by referring to valid ones. Considering the intrinsic property of depth maps, i.e., the sharpness of object boundaries, we propose to use arbitrary-shape matching patches, instead of fixed squares, to avoid inter-depth-layer distortion and thus improve the boundary. In addition, since depth values do not have distinct features, cross-modal matching, where both depth and texture are involved, is utilized. Moreover, we also investigate the similarity criteria in cross-modal matching, in order to improve the accuracy between the source patch and the target patch. Experimental results demonstrate that the proposed method can accurately recover lost depth information, especially at boundaries, which outperforms state-of-the-art exemplar-based inpainting methods.