Terpenoids, also known as isoprenoids, are the largest and most diverse class of organic compounds in nature and are involved in many membrane-associated cellular processes, including membrane organization, electron transport chain, cell signaling, and phototrophy. Terpenoids are ancient compounds with their origin presumably before the last universal common ancestor. However, Bacteria and Archaea are known to possess two distinct terpenoid repertoires and utilize terpenoids differently. Most notably, archaea constitute their cellular membrane solely made of terpenoid-based phospholipids, contrary to the bacterial membrane that consists of fatty acid-based phospholipids. Thus, the composition of ancestral membranes at the beginning of cellular life and the diversification of terpenoids in early life remain enigmatic. This review addresses these key issues through comprehensive phylogenomic analyses of extant terpenoid biosynthesis enzymes in Bacteria and Archaea. We aim to infer the basal components of terpenoid biosynthesis machinery that have an ancient origin before the divergence of the two domains and shed light on the deep evolutionary connection between terpenoid biochemistry and early life.