Impact of Fear and Group Defense on the Dynamics of a Predator–Prey System

To reduce the chance of predation, many prey species adopt group defense mechanisms. While it is commonly believed that such defense mechanisms lead to positive feedback on prey density, a closer observation reveals that it may impact the growth rate of species. This is because individuals invest more time and effort in defense rather than reproductive activities. In this study, we delve into a predator–prey system where predator-induced fear influences the birth rate of prey, and the prey species exhibit group defense mechanism. We adopt a nonmonotonic functional response to govern the predator–prey interaction, which effectively captures the group defense mechanism. We present a detailed mathematical analysis, encompassing the determination of feasible equilibria and their stability conditions. Through the analytical approach, we demonstrate the occurrence of Hopf and Bogdanov–Takens (BT) bifurcations. We observe two distinct types of bistabilities in the system: one between interior and predator-free equilibria, and another between limit cycle and predator-free equilibrium. Our findings reveal that the parameters associated with group defense and predator-induced fear play significant roles in the survival and extinction of populations.