Multimodal data fusion for object recognition

Multi-spectral imagery provides wide possibilities for improving quality of object detection and recognition due to better visibility of different scene features in different spectral ranges. To use the advantage of multi-spectral data the relation between different types of data is required. This relation is provided by capturing data using calibrated, aligned and synchronized sensors. Also geo-spatial data in form of geo-referenced digital terrain models can be used for establishing geometric and semantic relations between different types of data. The presented study considers the problem of object recognition based on two data sources: visible and thermal imagery. The main aim of the performed study was to evaluate the performance of different convolutional neural network models for multimodal object recognition. For this purpose a special dataset was collected. The dataset contains synchronized visible and thermal images acquired by several sensor based on unmanned aerial vehicle. The dataset contains synchronized color and thermal images of urban and suburb scenes gathered in different seasons, different times of day and various weather conditions. For convolutional neural network training the dataset was augmented by model images created using object 3D models textured by real visible and thermal images. Several convolutional neural network architectures were trained and evaluated on the created dataset using different splits to estimate the influence of training data on object recognition performance.