The thermoelectric effect allows direct and reversible conversion of thermal energy into electricity. As a result, thermoelectric generators and coolers can be an essential part of the solution to today’s energy challenges by reducing adverse effects on the environment. Nonetheless, further progress in thermoelectric research critically depends on designing novel thermoelectric materials that substantially exceed the current efficiency limits. As the thermoelectric performance is inversely proportional to the material's thermal conductivity, the design and discovery of materials with low thermal conductivity and robust electronic properties are of paramount importance. However, this quest for materials with low thermal conductivity is arguably the most challenging aspect of optimizing the thermoelectric modules. In this review, we first introduce the historical, experimental, and computational aspects of the concept of thermal conductivity. We then explore in detail the theoretical foundations of intrinsically low thermal conductivity in bulk and low-dimensional materials. We specifically examine how density functional and molecular dynamics calculations help identify low thermal conductivity characteristics such as bond anharmonicity, weak bonding of a rattling atom, cation disorder, and diffusion. Furthermore, we present high throughput computational screening strategies for discovering new materials with low thermal conductivity by discussing the recent advances in the relevant computational tools.