The valley-polarized quantum anomalous Hall effect (VP-QAHE), which combines valleytronics and topology in one material, is of significant fundamental and practical importance in condensed-matter physics and materials science. In previous model studies, VP-QAHE occurs under strong Rashba spin-orbit coupling (SOC), which is an extrinsic effect. Here, using a low energy $k\ifmmode\cdot\else\textperiodcentered\fi{}p$ model, we propose a different mechanism of VP-QAHE by introducing an intrinsic staggered magnetic exchange field and develop a general picture of valley dependent band inversion in honeycomb lattice. Using first-principles calculation, this new mechanism is further demonstrated in the Co decorated In-triangle adlayer on a Si(111) surface. This system is equivalent to a ferrimagnetic honeycomb lattice, and the supported adlayer is experimentally more feasible in synthesis, thus exhibiting advantages over the existing studies based on Rashba SOC and free-standing sheets. The underlying physical mechanism is generally applicable, opening a new avenue for exploration of substrate supported VP-QAHE.