The benchmark rate reform has significant impact on fixed income derivatives markets, and poses challenges in pricing risk-free rate (RFR) instruments. We propose a unified framework for pricing RFR caplets, including forward-looking, backward-looking and barrier caplets, under general Markov short rate models. Our approach is to approximate the RFR process using continuous-time Markov chain (CTMC). We derive an explicit expression of T-forward intensity and transition probability under CTMC for general interest rate derivatives, and use Laplace transform to price caplets with backward-looking rates. The framework offers a comprehensive and efficient approach to accurately price RFR caplets across various short rate models, and can be potentially extended to other types of RFR derivatives. Numerical experiments demonstrate the efficiency and accuracy of our pricing method under popular short rate models including the Vasicek model, CIR model, and jump-extended CEV model.