A predator-prey model with cooperative hunting in the predator and group defense in the prey

<p style='text-indent:20px;'>In this paper we propose a predator-prey model with a non-differentiable functional response in which the prey exhibits group defense and the predator exhibits cooperative hunting. There is a separatrix curve dividing the phase portrait. The species with initial population above the separatrix result in extinction of prey in finite time, and the species with initial population below it can coexist, oscillate sustainably or leave the prey surviving only. Detailed bifurcation analysis is carried out to explore the effect of cooperative hunting in the predator and aggregation in the prey on the existence and stability of the coexistence state as well as the dynamics of system. The model undergoes transcritical bifurcation, Hopf bifurcation, homoclinic (heteroclinic) bifurcation, saddle-node bifurcation, and Bogdanov-Takens bifurcation, and through numerical simulations it is found that it possesses rich dynamics including bubble loop of limit cycles, and open ended branch of periodic orbits disappearing through a homoclinic cycle or a loop of heteroclinic orbits. Also, a continuous transition of different types of Hopf branches are investigated which forms a global picture of Hopf bifurcation in the model.</p>