It has been generally accepted that unfilled skutterudites process high lattice thermal conductivity ($\kappa_{l}$) that can be efficiently reduced upon filling. Here by using first principles Boltzmann-Peierls transport calculations, we find pure skutterudite of FeSb$_3$ with no filler in fact has an intrinsic ultralow $\kappa_{l}$ smaller than that of CoSb$_3$ by one order of magnitude. The value is even smaller than those of most of the fully filled skutterudites. This finding means that with FeSb$_3$ as a reference, filling does not necessarily lower $\kappa_{l}$. The ultralow $\kappa_{l}$ of FeSb$_3$ is a consequence of much softened optical phonon branches associated with the weakly bonded Sb$_4$ rings. They overlap more with heat-carrying acoustic phonons and significantly increase the phase space for three-phonon anharmonic scattering processes. This provides an alternative non-filling related mechanism for lowering the $\kappa_{l}$ of skutterudites.