We propose a novel multi-stage constant false alarm rate (CFAR) detector for millimeter wave radars. In particular, we consider the frequency modulated continuous wave (FMCW) radars. First, we employ order statistics-based detector (OSD) on the range and Doppler dimensions, to obtain potential target locations as a coarse detection procedure. Next, we propose a weighted centroid detector (WCD) for fine detection on the range-Doppler matrix obtained from OSD, which is agnostic to the knowledge of noise variance. We obtain analytical expressions for the probabilities of false-alarm and detection threshold for both OSD and WCD, which are validated using Monte Carlo simulations. Through synthetic data and real-world experimental data, we highlight the efficacy of the proposed detectors in terms of the receiver operating characteristics and detection probability.