The thermoelectric properties of RbBaX (X = Sb, Bi), an anisotropic material with strong anharmonicity, are systematically studied by first-principles calculations, combined with the self-consistent phonon theory and the Boltzmann transport equation. A reasonable lattice thermal conductivity κL is captured by fully handling the phonon frequency shift and four-phonon scattering caused by the quartic anharmonicity. The κL of RbBaSb and RbBaBi along the a-axis is only 0.60 and 0.36 W m−1 K−1 at 300 K, respectively, which is much lower than that of most thermoelectric materials. The low phonon group velocity resulting from the unusually weak atomic bonding strengths along the a-axis is the origin of the ultralow κL. Furthermore, the high dispersion near the conduction band minimum enables n-type doping with a higher electrical conductivity. The results show that orthorhombic RbbaBi has a ZT as high as 1.04 at 700 K along the a-axis direction, indicating its great application potential in the thermoelectric field.