In recent years, the 3D NAND stack has continued to expand rapidly in the Z direction with tighter process windows, while deck-to-deck overlay faces increasing measurability and accuracy challenges. Meanwhile, thick opaque hard mask (HM) has been utilized in several layers, thus downgrades a lot of mark contrast and introduces large process variation (PV). To meet tightened overlay requirements, KLA’s Metrology Target Design (MTD) assisted by simulation analysis has become an important part of the holistic overlay improvement solution set. Compared to the conventional target design simulation process in which the target stack is built up with a standard template, in this work, a customized template was designed to fit actual mark behavior. Using this new template, more accurate simulation to measurement (S2M) matching was demonstrated, which makes the new target design more reliable. With this added benefit, the newly designed target achieved desirable residual improvement with better on-product overlay (OPO) performance. In the case of the challenging thick opaque HM layer, target segmentation and CD/Pitch optimization by simulation also significantly improved mark contrast and OPO using the newly designed mark. Along with contrast gain, the robustness against large PV of the simulated mark was increased by quantifying comprehensive PV on product wafers. Furthermore, with the verification of the measured data, simulation data can be used to establish a more thorough representation of the process characteristics and parametric sensitive virtual metrology, targeted to meet the goals of maximizing overlay accuracy in the 3D NAND process.