It has been established that both hybridization with magnetic rare earth cation and crystal field originating from structural anisotropy can affect the spin orientation of magnetic transition metal perovskite. In this work, we have systematically studied the crystal structure, magnetization, and magnetic structure of Mn-doped rare earth orthochromites HoCr1−xMnxO3 (x = 0–0.85) by means of X-ray diffraction, magnetometry, and neutron powder diffraction. With increasing Jahn–Teller active Mn3+-substitution for Cr3+, the crystallographic distortion is gradually enhanced and causes a systematic evolution of magnetism. With doping, the Cr(Mn) ordering temperature is gradually suppressed, and multiple magnetic configurations Γ2, Γ1, and Γ4 phases appear consecutively. The rich transition phenomena are ascribed to a competition between increased structural anisotropy and weakened Ho-Cr(Mn) hybridization with doping. In a low doping regime with x = 0.1–0.5, Γ2 phase is predominant in low temperature, and a magnetization reversal is observed, reflecting the opposite sign of Cr-Mn DMI coefficient with respect to the Cr-Cr and Mn-Mn interaction. In the high doping range with x = 0.6–0.8, the long-range order of Cr(Mn) sublattice is gradually suppressed while a short-range order becomes dominant. For x = 0.85, a magnetic structure resembling o-HoMnO3 is observed.