The inverse design of metamaterials is the mapping of spectral response (low dimensions) to the structure (high dimensions). In the process of inverse design, there will be a "one-to-many" problem (similar spectral responses are generated by multiple meta-atoms), especially for designs with high degrees of freedom (DOFs). Although traditional generative models (VAE, GAN) combined with optimization algorithms can convert inverse design into forward prediction to solve the "one-to-many" problem, this method requires multiple iterative calculations and has low design efficiency, making it difficult to achieve fast and accurate inverse design. This paper proposes a novel inverse design method based on the image-parameter diffusion model. Through a step-by-step process of adding noise that transforms the structural parameters and images into Gaussian distributions, our method can gradually remove the noise starting from the distribution and generate high DOFs meta-atoms that satisfy the transmission coefficient of additional inputs. Moreover, this method avoids multiple iterative optimizations and ensures more efficient and high-quality generation results. The experimental results show that our method solves the "one-to-many" problem of high-DOFs meta-atoms and can generate free-form structures with only one design iteration, which is superior to conventional design methods in terms of model generation speed, generation accuracy, and quality.