A fast object classification scheme in terahertz (THz) domain is proposed. This scheme combines machine learning and THz single-pixel imaging (SPI) technologies but skips the image reconstruction in SPI. It can directly perform the linear pattern recognition task based on a few single-pixel values. This approach has advantages of low experimental complexity, simple optimization algorithms, and all-optical processing. It is verified in simulation and experiment with a classification accuracy of 90%. Our proposed scheme paves an avenue for fast classification of THz images in practical applications.