First-principles study of magnetic states and the anomalous Hall conductivity of MNb3S6 (M=Co,

Inspired by the observation of the extremely large anomalous Hall effect in the absence of applied magnetic fields or uniform magnetization in ${\mathrm{CoNb}}_{3}{\mathrm{S}}_{6}$ [Nat. Commun. 9, 3280 (2018); Phys. Rev. Research 2, 023051 (2020)], we perform a first-principles study of this and related compounds of the $M{\mathrm{Nb}}_{3}{\mathrm{S}}_{6}$ type with different transition metal $M$ ions to determine their magnetic orders and the anomalous Hall conductivity (AHC). We find that noncoplanar antiferromagnetic ordering is favored relative to collinear or coplanar order in the case of $M$ = Co, Fe, and Ni, while ferromagnetic ordering is favored in ${\mathrm{MnNb}}_{3}{\mathrm{S}}_{6}$ at low temperatures. The AHC in these materials with noncoplanar spin ordering can reach about ${e}^{2}/h$ per crystalline layer, while being negligible for coplanar and collinear cases. We also find that the AHC depends sensitively on doping and reaches a maximum for intermediate values of the local spin exchange potential between 0.3 and 0.8 eV. Our AHC results are consistent with the reported Hall measurements in ${\mathrm{CoNb}}_{3}{\mathrm{S}}_{6}$ and suggest a possibility of similarly large anomalous Hall effects in related compounds.