The changes in the crystallographic parameters of Y0.79-zAlzYb0.18Er0.03NbO4 powders prepared with varying Al content were determined by Rietveld refinement, and their effects on the up- (UC) and down-conversion (DC) spectra were studied. The findings confirmed that the Al3+, Yb3+, and Er3+ ions were completely incorporated at the Y3+ sites. The substitution of the small Al3+ ions led to the modification of the interatomic distance. As a result, the highest polyhedral distortion index of the YO8 dodecahedron (DYO) could be achieved at z = 0.2. Under infrared radiation, UC spectra demonstrated that a two-photon energy transfer (ET) UC process from Yb3+ to Er3+ was responsible for the green and red emissions, whose intensities increased dramatically by 160 and 166% at z = 0.2, respectively. This enhancement was a consequence of a large crystal-field asymmetry surrounding the Yb3+ and Er3+ ions, which resulted from the highest DYO at z = 0.2. In addition to D, the Yb‒Er interatomic distance was responsible for the enhanced UC emission, affecting the ET probability from Yb3+ to Er3+. Under 379 nm excitation, the DC green emission of Er3+ also showed the maximal intensity at z = 0.2 owing to the largest DYO. However, a correlation between the crystallographic variations and the DC emissions activated by [NbO4]3- niobates could not be fully elucidated owing to the complexity of the emission mechanism.