Nanoparticle-on-mirror plasmonic nanocavities, capable of extreme optical confinement and enhancement, have triggered state-of-the-art progress in nanophotonics and development of applications in enhanced spectroscopies and molecular detection. However, the optical quality factor and thus performance of these nanoconstructs are undermined by the granular polycrystalline metal films used as a mirror. Here, we report an atomically smooth single-crystalline platform for low-loss nanocavities using chemically-synthesized gold microflakes as a mirror. Nanocavities constructed using gold nanorods on such microflakes exhibit a rich structure of plasmonic modes, which are highly sensitive to the thickness of optically-thin (down to ~15 nm) microflakes. The atomically smooth single-crystalline microflakes endow nanocavities with significantly improved quality factor (~2 times) and scattering intensity (~3 times) compared with their counterparts based on deposited films. The developed low-loss nanocavities further allow for the integration with a mature platform of fiber optics, opening opportunities for realizing nanocavity-based miniaturized photonic devices with high performance.