With the advent of high-resolution remote sensing images, automatic building extraction methods play a more important role in rapidly acquiring information about large-scale buildings. Although advanced building extraction methods have been introduced to improve building extraction results, these methods involve complex processing and high-computation times. We put forward an effective method to extract building information, based on a proposed spectral building index. The basic idea of the spectral building index is to generate an optimized index based on the computation and analysis of spectral bands, which are beneficial for image enhancement for buildings in images. Aiming at the band number of the multispectral satellite images in high-resolution remote sensing images, we propose two spectral indices for building extraction, including the normalized spectral building index (NSBI) and the difference spectral building index (DSBI). Considering the current spectral band number of high-resolution satellite images, NSBI is suited for satellite images with eight spectral bands, whereas DSBI is suited for satellite images with four spectral bands. The proposed method is validated on various high-resolution images including WorldView-2, GF-1, GF-2, and QuickBird images with 13 experiment datasets, as well as a detailed comparison to the state-of-the-art methods, such as the morphological building index, nonhomogeneous feature difference, and building condition index. The experimental results reveal that the proposed method can achieve promising results for different building conditions, such as regular and irregular building shapes and concrete and metal roofing materials. The average overall accuracy was over 85% with low-time consumption (<1 s).