This article is concerned with the effects of time-varying impulses on exponential stability to a unique equilibrium point of inertial Bidirectional Associative Memories (BAM) neural network with mixed time-varying delays. A suitable variable transformation is chosen to transform the original system into a system of first order differential equations. The concept of homeomorphism has been implemented to find a distributed delay-dependent sufficient condition which assures that the system has a unique equilibrium point. In order to study the impulsive effects on stability problems, a time-varying impulses, including stabilizing and destabilizing impulses, are considered with the transformed system. Based on the matrix measure approach and an extended impulsive differential inequality for a time-varying delayed system, we have derived sufficient criteria in matrix measure form which ensure the exponential stability of the system towards an equilibrium point for two classes of activation functions. Further, different convergence rates of the system’s trajectory have been discussed for the cases of time-varying stabilizing and destabilizing impulses using the concept of an average impulsive interval. Finally, the efficiency of the theoretical results has been illustrated by providing two numerical examples.