RGB-D Gate-guided edge distillation for indoor semantic segmentation

Fusing the RGB and depth information can significantly improve the performance of semantic segmentation since the depth data represents the geometric information. In this paper, we propose a novel Gate-guided Edge Distillation (GED) based approach to effectively generate edge-aware features by fusing the RGB and depth data, assisting the high-level semantic prediction. The proposed GED consists of two modules: gated fusion and edge distillation. The gated fusion module adaptively learns the relationship between RGB and depth data to generate complementary features. To address the adverse effects caused by redundant information of edge-aware features, edge distillation module enhances the semantic features of the same object while preserving the discrimination of the semantic features belonging to different objects. Besides, by using distilled edge-aware features as detailed guidance, the proposed edge-guided fusion module effectively fuses with semantic features. In addition, the complementary features are leveraged in multi-level feature fusion module to further enhance detailed information. Extensive experiments on the widely used SUN-RGBD and NYU-Dv2 datasets demonstrate that the proposed approach with ResNet-50 achieves state-of-the-art performance.