Moiré fringe has become one of the mainstream methods for high-precision distance measurement at present, and the accuracy of moiré fringe-based distance measurement can reach the nanometer level. However, due to the discrete sampling of moiré fringe images by camera and the non-integer sampling, spectral leakage occurs during Fourier transform, thereby affecting the measurement accuracy. By applying a 2D-windowing process to the moiré fringe, the spectral leakage can be effectively reduced and the accuracy of the moiré fringe phase demodulation improved. This paper conducted a simulation analysis on moiré fringes formed by three sets of gratings with different periods and obtained the most suitable window function for each of them. Finally, one set of the gratings was selected for height displacement measurement experiments on a wafer stage. The experimental results achieved a measurement accuracy better than 5nm.