Abstract This study investigates the vibrational properties of hydroxyl groups in pure and Fe-bearing brucite under high pressure and temperature conditions relevant to subduction zones. Using externally-heated diamond-anvil cell techniques combined with in-situ laser Raman spectroscopy, we observed that the hydroxyl stretching modes in pure brucite change from blueshift to redshift at approximately 600 K and 1.2 GPa. This indicates a sublattice proton order-disorder transition associated with decreasing interlayer spacing in brucite. Moreover, the Fe-Mg substitution in brucite leads to a secondary hydroxyl stretching band at 3644.6 cm−1 for Mg0.9Fe0.1(OH)2 while the primary Raman band is changed to 3636.7 cm−1. This new Raman mode persists with increasing temperature at atmospheric pressure, but disappears at about 3.0 GPa. The decomposition temperature of this Fe-bearing brucite sample is about 130 K lower compared to pure brucite under high oxygen fugacity conditions. These results suggest that the sublattice transition and Fe-Mg substitution may influence the stability of brucite, providing insight into the deep-water cycle at shallow subduction zones.