Nowadays, microwave dielectric substrate materials have been extensively investigated to meet the requirements of rapid development in modern communications. Among them, the composites of ceramic powder filled polytetrafluoroethylene (PTFE) have been a hot topic. However, the compatibility and connectivity between the surface of ceramics and PTFE molecular chains in the samples are usually low. Herein novel PTFE based composites with different contents of Al2Mo3O12 (20–60 wt%) modified by C14H19F13O3Si (F8261) coupling agent were designed and prepared. The coupling agent F8261 has been successfully grafted to the surface of Al2Mo3O12 powders, effectively promoting the densification and dielectric properties of the composites. As the content of the modified Al2Mo3O12 powders increases from 20 to 60 wt%, the εr value increases from 3.4 to 4.2, and tanδ almost remains constant at the beginning and increases with much more Al2Mo3O12 added. The Al2Mo3O12-PTFE composites filled with 30 wt% Al2Mo3O12 present the optimal dielectric properties of εr = 3.6 and tanδ = 0.0018 with a high density of 95.6%. In addition, the electromagnetic and multiphysic simulation of a 24 GHz substrate integrated waveguide filter on the basis of the 30 wt% Al2Mo3O12 - 70 wt% PTFE composite was carried out. It was revealed that the filter presented high stability on the electrical parameters caused by self-heating and dimension deformation due to the good microwave dielectric, thermal and mechanical properties of the substrate. These results indicate that the as-prepared 30 wt% Al2Mo3O12 - 70 wt% PTFE composite would be a promising candidate for high-performance microwave dielectric substrates.