Abstract Off‐centering represents the displacement of atoms from their centrosymmetric positions, a phenomenon closely associated with symmetry breaking in crystals and plays a crucial role in modulating a broad range of fundamental material properties. Herein, a pavonite‐structured compound Ag 1.75 InSb 5.75 Se 11 (AISS) is reported, which exhibits highly off‐centering cations and systematically explores its potential as an advanced thermoelectric material. These findings reveal that the broken crystal symmetry of AISS induces a strong Rashba‐like band splitting, resulting in an overall high band degeneracy that contributes to large Seebeck coefficients. Additionally, low‐temperature heat capacity measurements and phonon spectrum calculations highlight the presence of soft chemical bonds between the off‐centering cations and Se atoms. These soft bonds give rise to low‐frequency optical phonons, which coupled with the acoustic phonons, thereby causing additional phonon scattering and leading to an extremely low lattice thermal conductivity of 0.22 W m −1 K −1 at 723 K. Furthermore, optimizing the Ag/Sb atomic ratio in AISS enhances the doping level and improves the thermoelectric performance, resulting in a peak figure of merit value to 0.50 at 723 K in Ag 2.5 InSb 5 Se 11 , which is ≈30 times greater than that of pristine AISS.