We observe an elusive defect in face detectors which is overlooked in existing works. Specifically, we find that face detectors output unstable confidence scores when faces are slightly shifted in position. The confidence scores of the shifted faces can be lower than the detection threshold and result in false negatives. We define this phenomenon as face detection spatial instability. In essence, detectors face the spatial instability problem because they perform badly for challenging object positions. An object position is challenging when it is not sufficiently modeled by any existing anchors. To deal with this problem, we propose Matching Similarity Auxiliary (MSA) box, which consists of three parts: MSA assign, Image Digitization Compensator (IDC), and Soft Smooth L1 Loss (SSL-Loss). Specifically, MSA assign and IDC are designed to dig more challenging samples, which guide the face detectors to enhance detection performance in these extreme cases. SSL-Loss balances the training phase based on the regression efforts of each target. It improves the localization performance even with more challenging positive samples involved. Experiments demonstrate that our MSAbox achieves spatially stable face detection.