The anisotropic removal behavior of single crystal calcium fluoride (CaF2) severely restricts the improvement of surface quality in ultra-precision machining. In this study, the anisotropic mechanical removal of CaF2 material was investigated quantitatively along various crystal orientations on CaF2(111), (110), and (001) surfaces by a conical diamond tip. With the increase of load, the wear of CaF2 material has an elastic–ductile–brittle transition, and the corresponding critical load and wear depth strongly depend on the basis of crystal plane and orientation. Plastic deformation initially occurs in the directions of mechanical stress coinciding with the slip system of CaF2 material, and the crack grows preferentially on the cleavage plane. The contact stress range of 10.8 GPa to 11.4 GPa is defined as the security region in which the material removal of CaF2 can be controlled in the ductile wear under all crystal plane and orientation conditions. The results may help to optimize the processing parameters to achieve consistent and controllable material removal in the fabrication of CaF2 with complicated shapes.