Warfarin Dose Management Using Offline Deep Reinforcement Learning

Warfarin is a commonly prescribed anticoagulant with a narrow therapeutic window, requiring frequent and specialized monitoring. This work aims to develop standardized optimal warfarin dose decision support using a machine learning model based on time series anticoagulation data and patient demographic characteristics. We propose an offline reinforcement learning model (RL) using a Batch-Constrained Q-Learning algorithm (BCQ) in the discrete action setting to predict the cumulative warfarin dose for the days until the next INR (International Normalized Ratio) test. Prior approaches utilized time-series supervised learning methods such as regression or Long Short Term Memory (LSTM) neural networks. The key advantage of reinforcement learning is its capacity to learn optimal dosing strategies from suboptimal clinical states in the data. To evaluate the model we compared the predicted warfarin doses with the physician-prescribed doses. Our BCQ model with a prediction accuracy of 98.6% significantly outperformed our baseline Long Short Term Memory (LSTM) model with a prediction accuracy of 71.09%. Further qualitative evaluation for explainability indicated that the model correctly adjusted the warfarin dose at time steps when patients had out-of-range INRs.