The coupling of the lattice, charge, spin, and valley degrees of freedom is a fundamental scientific issue in spintronics and valleytronics. The nonvolatile electric field control of the spin and valley polarization has important potential applications for the development of next-generation ultrahigh speed processors and memories. However, the efficient electrical control of spin and valley degrees of freedom remains a challenge due to the weak coupling between them and external electric fields. Here we report the strong coupling between the spin, valley, and electric polarization in a two- dimensional multiferroic material ${\mathrm{Nb}}_{3}{\mathrm{I}}_{8}$ with a breathing Kagome lattice. The ferroelectric transition controls the local sublattice symmetry in the ferroelectric monolayer ${\mathrm{Nb}}_{3}{\mathrm{I}}_{8}$ and thus results in the sublattice-dependent Berry curvature switching in the momentum space, where the sign reversal of the Berry curvature can be controlled by reversing the ferroelectric polarization. More importantly, for the ferroelectric bilayer ${\mathrm{Nb}}_{3}{\mathrm{I}}_{8}$ with A-type antiferromagnetic coupling, the spin-valley-layer polarized anomalous Hall effect can be realized by coupling the electric polarization to spin, valley and layer degrees of freedom. So, the electrically driven the ferroelectric transition in a ${\mathrm{Nb}}_{3}{\mathrm{I}}_{8}$ bilayer can be applied to design nonvolatile memory and switch based on the spin, valley and layer dependent Berry curvature. Our findings open an avenue towards exploring the coupling between the ferroelectricity, ferromagnetism, and ferrovalley in the hidden local sublattice symmetry and demonstrate a nonvolatile electric-field controlled anomalous Hall effect in the atomically thin limit.