We present a novel design method of freeform imaging spectrometers. All the freeform surfaces including the grating surface are generated point-by-point. The rays coming from multiple wavelengths, multiple fields and different pupil coordinates are used, and the refractive/reflective as well as the diffraction of grating are considered simultaneously. To demonstrate the feasibility of the proposed method, we successfully designed a compact freeform reflective imaging spectrometer.