A compact varifocal panoramic annular lens (PAL) system based on the four-component mechanical zoom method is proposed, which solves the problem that the traditional PAL system cannot zoom in to the region of interest. By moving the zoom group and the compensation group, our design achieves continuous zooming, in which the focal length changes from 3.8 to 6 mm. It can keep the position of the image surface unchanged while maintaining a compact structure. The system has a field of view (FoV) of 25°–100° in wide-angle mode and an FoV of 25°–65° in telephoto mode. The modulation transfer function of the wide-angle view is higher than 0.22 at 147 lp/mm. The F -theta distortion is less than 3%, and the relative illuminance is higher than 0.9 in the zoom process. Compared with the zoom PAL system with multiple free-form aspheric surfaces, the proposed system uses multiple spherical lenses and only one Q -type asphere lens to achieve outstanding panoramic zoom imaging results. It is practical and straightforward, easy to manufacture, detect, and mass produce.