Coherent modulation schemes, such as QPSK, have emerged as promising candidates for future short-range optical communications to meet the rapidly rising demand of data center traffics. To evaluate and compare the performance of QPSK with conventional amplitude modulation schemes including NRZ, PAM-4 and PAM-8, a VCSEL-based optical link is presented in this paper, including transceiver front-end circuits, models of VCSEL, multi-mode fiber and photodiode. An Integrated System Evaluation Engine (ISEE) is implemented to empower the cross-domain simulation using advanced modulation signals. The simulation results reveal the relationship between the ratio of input symbol rate over carrier frequency and data rate for QPSK, as well as the optimal configuration of feed-forward equalizer (FFE) for NRZ, PAM-4 and PAM-8. The comparison demonstrates that QPSK outperforms NRZ in terms of both data rate and tolerance to poor SNR.