We intentionally designed a stable two-dimensional (2D) bilayer dimer system, OsFeP4, which serves as a tunable spin-5/2 J1-J2 anisotropic Heisenberg-like model to investigate frustrated magnetism. The frustrated state of 2D OsFeP4 originates from the robust intradimer ferromagnetic (FM) exchange coupling (J⊥) and a subtle competition between P4 molecules mediating interdimer ferromagnetic (J∥) and antiferromagnetic (AFM) exchange coupling (J×). This is further supported by a significant broadening of the specific heat peak, a high frustration factor (θCW/TN) of 4.17, the peak at the gamma point in the static structural factor, and short-range spin textures. Moreover, we employ strain to modulate the frustration parameter J×/J∥ from 0.23 to 0.4 and spin direction from in-plane to out-of-plane. The FM, AFM, and frustrated phase transitions are identified in the J⊥-J×-J∥ ternary phase diagram. This work will enrich our understanding of the frustration mechanisms in the 2D limit.