Magnetic properties and spin dynamics in the spin-orbit driven Jeff=12 triangular lattice antiferromagnet Ba6Yb2

Frustration-induced strong quantum fluctuations accompanied by spin-orbit coupling and crystal electric field can give rise to rich and diverse magnetic phenomena associated with unconventional low-energy excitations in rare-earth-based quantum magnets. Herein, we present crystal structure, magnetic susceptibility, specific heat, muon spin relaxation ($\textmu{}\mathrm{SR}$), and electron spin resonance (ESR) studies on polycrystalline samples of ${\mathrm{Ba}}_{6}{\mathrm{Yb}}_{2}{\mathrm{Ti}}_{4}{\mathrm{O}}_{17}$, in which ${\mathrm{Yb}}^{3+}$ ions constitute a perfect triangular lattice in the $ab$ plane without detectable antisite disorder between atomic sites. The Curie-Weiss fit of the low-temperature magnetic susceptibility data suggests spin-orbit driven effective pseudospin ${J}_{\mathrm{eff}}=\frac{1}{2}$ degrees of freedom of ${\mathrm{Yb}}^{3+}$ spin with weak antiferromagnetic exchange interactions in the Kramers doublet ground state. The zero-field specific heat data reveal the presence of long-range magnetic order at N\'eel temperature ${T}_{\mathrm{N}}=77\phantom{\rule{0.28em}{0ex}}\mathrm{mK}$ which is suppressed in a magnetic field ${\ensuremath{\mu}}_{0}H\ensuremath{\ge}1\phantom{\rule{0.28em}{0ex}}\mathrm{T}$. The broad maximum in specific heat is attributed to the Schottky anomaly implying the Zeeman splitting of the Kramers doublet ground state in a magnetic field. The ESR measurements suggest the presence of anisotropic exchange interaction between the moments of ${\mathrm{Yb}}^{3+}$ spins. The $\textmu{}\mathrm{SR}$ experiments reveal a fluctuating state of ${\mathrm{Yb}}^{3+}$ spins in the temperature range $0.1\phantom{\rule{0.28em}{0ex}}\mathrm{K}\ensuremath{\le}T\ensuremath{\le}10\phantom{\rule{0.28em}{0ex}}\mathrm{K}$ owing to depopulation of crystal electric field levels, which suggests that the lowest Kramers doublets with ${J}_{\mathrm{eff}}=\frac{1}{2}$ are well separated, and the low-temperature physics of this frustrated magnet is dominated by ${J}_{\mathrm{eff}}=\frac{1}{2}$ moments. In addition to the intraplane nearest-neighbor superexchange interaction, the interplane exchange interaction and anisotropy are expected to stabilize the long-range ordered state in this triangular lattice antiferromagnet.