Real-time CNN-based keypoint detector with Sobel filter and descriptor trained with keypoint candidates

In this paper, we propose two separate and lightweight convolutional neural networks, SobelNet and DesNet, which work in parallel, as keypoint detector and descriptor respectively. Sobel filter provides the edge structure map of the grayscale image as the input of SobelNet. The locations of keypoints will be obtained after exerting the non-maximum suppression process on the output score map of SobelNet. Gaussian loss is designed to train SobelNet to detect corner points in the edge structure map as keypoints. In the meantime, a dense descriptor map is produced by DesNet which is trained with Circle loss. Besides, the output score map of SobelNet is utilized while training DesNet. The proposed method is evaluated on two widely used datasets, FM benchmark and ETH benchmark. Compared with other state-ofthe- art methods, SobelNet and DesNet can reduce more than half of the computation and achieve comparable or even better performance. The inference times of an image with the size of 640×480 are 7.59 ms and 1.09 ms for SobelNet and DesNet respectively on RTX 2070 SUPER, which meet the real-time requirement.