Herbal medicines have been applied in clinical treatment worldwide, whose significant curative effect attracts considerable global research attention. However, as the herbal medicine industry develops continuously in recent years, new challenges including monitoring the quality and safety of herbal medicines appear in this industry. Numerous cases of fungal and mycotoxin contamination have been reported that affected the quality and safety of herbal medicines. The main mycotoxins found in herbal medicines include aflatoxin, ochratoxin A, and fumonisin B, which cause substantial harm to human health. They are mainly produced by species from Aspergillus, Penicillium, and Fusarium. Various reports have focused on studying the conditions for fungal growth and mycotoxin synthesis to provide references for prevention. The chemical compounds and antagonism microorganisms were also explored to inhibit fungal growth, and decrease mycotoxin accumulation. This review discusses natural occurrence of three main fungal genera (Aspergillus, Penicillium, and Fusarium) and three main mycotoxins (aflatoxin, ochratoxin A, and fumonisin B) in herbal medicines, analyzing the endogenous and exogenous factors that affect fungal growth and mycotoxin production. Moreover, the prevention methods of fungal contamination are included.