Thermally conductive polymer nanocomposites are enticing candidates for not only thermal managements in electronics but also functional components in emerging thermal energy storage and conversion systems and intelligent devices. A high thermal conductivity (k) depends largely on the ordered assembly of high-k fillers in the composites. In the past decades, various templating assembly techniques have been developed to rationally construct nanoscale fillers into three-dimensional (3D) interconnected structures, further improving the k of composites compared to conventional methods. Herein, recent advances are summarized in developing thermally conductive polymer composites based on self-templating, sacrificial templating, foam-templating, ice-templating and template-directed chemical vapor deposition techniques. These unique templating methods to fabricate 3D interconnected fillers in the form of segregated, cellular, lamellar, and radially aligned structures are reviewed, and their correlations to the k of composites are thoroughly probed. Moreover, multiscale structural design strategies combined with different templating methods to further improve the k of composites are highlighted. This review offers a constructive guidance to fabricate next-generation thermally conductive polymer composites for diverse thermal energy applications.