Validation of an Electronic Health Record-Based Machine Learning Model Compared to Clinical Risk Scores for Gastrointestinal Bleeding

Abstract

Background & Aims

Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB.

Methods

The training cohort comprised 2,546 patients and internal validation of 850 patients presenting with overt GIB (hematemesis, melena, hematochezia) to emergency departments of 2 hospitals from 2014-2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014-2019. The primary outcome was a composite of red-blood-cell transfusion, hemostatic intervention (endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR available within 4 hours of presentation and compared performance of machine learning models to current guideline-recommended risk scores, Glasgow-Blatchford Score (GBS) and Oakland Score. Primary analysis was area under the receiver-operating-characteristic curve (AUC). Secondary analysis was specificity at 99% sensitivity to assess proportion of patients correctly identified as very-low-risk.

Results

The machine learning model outperformed the GBS (AUC=0.92 vs. 0.89;p<0.001) and Oakland score (AUC=0.92 vs. 0.89;p<0.001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs. 18.5% for GBS and 11.7% for Oakland score (p<0.001 for both comparisons).

Conclusions

An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department.