Selective Progressive Learning for Sparse Depth Completion

In this paper, we propose selective progressive learning (SPL) for sparse depth completion. Given an intermediate feature map selected from the input RGB and sparse depth image pairs by the valid and invalid mask, SPL sequentially infers attention maps along two separate dimensions (spatial attention and channel attention), and then the attention maps are used to extract features for depth regression. First, we design sparsity selection module (SSM) to simplify the task and segment it as depth prediction from sparse depth guided by corresponding RGB pixels and RGB to depth translation. Second, we propose a guided attention module (GAM) to translate features from SSM to attention maps by spatial and channel transformations. We use GAM as guidance for two pathways in the depth prediction network. Third, we combine two pathways by three cascade regressions with quarter-size, half-size and full-size of the input. We use multi-scale learning to maximize the advantage of cascade regressions, which utilizes down-sampling to get the ground truth of their corresponding size. The SPL network structure based on sparsity greatly reduces the amount of calculation, and SPL successfully bridges the modality gap between RGB and depth images for sparse depth completion. Experimental results show that SPL generates competitive dense depth images with low complexity and outperforms state-of-the-art ones in terms of both accuracy and robustness. Ablation experiments demonstrate the generalization capability and stability of SPL.