To understand the quality characteristics of different varieties of starch granules from sweet potato and their relationship with processing adaptability, 21 typical sweet potato varieties cultivated at Zhejiang A&F University are used to study grain size using a Mastersizer 3000 laser particle size analyzer. Our results are as follows: 1) The volume distribution of starch granules is a bimodal curve and the grain size is 2.27 μm. Starch granules with a particle size <2.27 μm are defined as small starch granules, starch granules with a particle size of 2.27–17.51 μm are defined as medium starch granules, and starch granules with a particle size >17.51 μm are defined as large starch granules. 2) The contents of total starch, amylose, and amylopectin are negatively correlated with the volume percentage of middle-sized (r = −0.611**, −0.489**, and −0.529**) starch granules but positively correlated with large-sized (r = 0.628**, 0.544**, and 0.517**) starch granules. 3) In processing applications of sweet potato, the cohesiveness of dried sweet potato is improved and hardness of dried sweet potato decreases with the increase of small- and medium-sized starch granules content. The color, fragrance, exterior, and taste of sweet potato chips are improved with the increase of small- and medium-sized starch granule content. Above results could help understand relationships between granule size and final product quality.