Deciphering the ecological role of soil communities in the maintenance of multiple ecosystem functions is pivotal for the conservation and sustainability of soil biodiversity. However, few studies have investigated niche differentiation of abundant and rare microbiota, as well as their contributions to multiple soil elemental cycles, particularly in agroecosystems that have received decades of intense fertilization. Here, we characterized the environmental thresholds and phylogenetic signals for the environmental adaptation of both abundant and rare microbial subcommunities via amplicon sequencing and metagenomic sequencing and explored their importance in sustaining soil multiple nutrient cycling in agricultural fields that were fertilized for two decades. The results showed that rare taxa exhibited narrower niche breadths and weaker phylogenetic signals than abundant species. The assembly of abundant subcommunity was shaped predominantly by dispersal limitation (explained 71.1 % of the variation in bacteria) and undominated processes (explained 75 % of the variation in fungi), whereas the assembly of rare subcommunity was dominated by homogeneous selection process (explained 100 % of the variation in bacteria and 60 % of the variation in fungi). Soil ammonia nitrogen was the leading factor mediating the balance between stochastic and deterministic processes in both abundant (R2 = 0.15, P < 0.001) and rare (R2 = 0.08, P < 0.001) bacterial communities. Notably, the rare biosphere largely contributed to key soil processes such as carbon (R2bacteria = 0.03, P < 0.05; R2fungi = 0.05, P < 0.05) and nitrogen (R2bacteria = 0.03, P < 0.05; R2fungi = 0.17, P < 0.001) cycling. Collectively, these findings facilitate our understanding of the maintenance of rhizosphere bacterial and fungal diversity in response to agricultural fertilization and highlight the key role of rare taxa in sustaining agricultural ecosystem functions.