Drilling fluids play crucial role in the oil and gas production industry, consequently the optimization of flow behavior of these fluids has received remarkable attention from the beginning of this process. The objective of this study is to improve the rheological and filtration characteristics of water-based mud (WBM) by incorporating zinc oxide (ZnO) nanoparticles (NPs) and affordable tragacanth gum (TG) polymer. Initially, ZnO NPs were synthesized through a facile and straightforward sol–gel method and subjected to comprehensive characterization. Subsequently, these NPs were incorporated into TG polymer solutions to formulate drilling mud samples. The selection of optimal samples involved meticulously determining of ZnO NPs and TG polymer concentrations as influential parameters facilitated by the Design Expert software. The responses taken into consideration in this methodology included flow behavior index (n), consistency index (K), plastic viscosity (PV), and yield point (YP). The investigating rheological parameters entailed utilizing both the Power-law and Bingham-plastic models, underscoring the scholarly refinement of the study's analytical approach. Subsequently, an analysis of variance (ANOVA) was conducted to assess the significance of the model and its parameters. Three optimal samples were carefully selected based on the results obtained from the modeling and analysis of the rheological parameters. The next phase involved utilizing these optimal samples to investigate their filtration properties in detail. The final step led to selecting the optimal sample composition, which was determined to be 0.25 wt% ZnO NPs and 750 ppm TG polymer based on the optimization results provided by the Design Expert software. Remarkably, this specific combination yielded significant improvements in PV (22.2 %), YP (109.5 %), mud cake thickness (20 %), and fluid loss (2.87 %). Moreover, the resulting mud exhibited desirable rheological properties, including a K of 31.20 (cp) and a n of 0.21. This specific combination was identified as the most favorable composition for achieving the desired rheological and filtration properties.