The utilization of optical coherence tomography (OCT) holds significant promise in the realm of cervical cancer screening. This study considers the characteristics of OCT, such as sample scarcity and noise, and introduces a noise adaptive diffusion model (NADM) for data augmentation. The NADM comprises two components: the Adaptive Blind De-noising Module (ABD) and the Class-Guided Diffusion Model (CDM). The CDM is responsible for generating high-quality cervical samples, while the ABD is designed to adaptively suppress noise. Extensive experiments are done to establish the performance and generalizability of NADM, based on data collected from three distinct hospitals. To the best of our knowledge, we are the first to employ a data augmentation technique utilizing generative models for the purpose of diagnosing cervical OCT images.