The identification and categorization of subsurface damages in thermal images of concrete structures remain an ongoing challenge that demands expert knowledge. Consequently, creating a substantial number of annotated samples for training deep neural networks poses a significant issue. Artificial intelligence (AI) models particularly encounter the problem of false positives arising from thermal patterns on concrete surfaces that do not correspond to subsurface damages. Such false detections would be easily identifiable in visible images, underscoring the advantage of possessing additional information about the sample surface through visible imaging. In light of these challenges, this study proposes an approach that employs a few-shot learning method known as the Siamese Neural Network (SNN), to frame the problem of subsurface delamination detection in concrete structures as a multi-modal similarity region comparison problem. The proposed procedure is evaluated using a dataset comprising 500 registered pairs of infrared and visible images captured in various infrastructure scenarios. Our findings indicate that leveraging prior knowledge regarding the similarity between visible and thermal data can significantly reduce the rate of false positive detection by AI models in thermal images.