Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules

Guidelines recommend risk stratification scores in patients presenting with gastrointestinal bleeding (GIB), but such scores are uncommonly employed in practice. Automation and deployment of risk stratification scores in real time within electronic health records (EHRs) would overcome a major impediment. This requires an automated mechanism to accurately identify ("phenotype") patients with GIB at the time of presentation. The goal is to identify patients with acute GIB by developing and evaluating EHR-based phenotyping algorithms for emergency department (ED) patients.We specified criteria using structured data elements to create rules for identifying patients and also developed multiple natural language processing (NLP)-based approaches for automated phenotyping of patients, tested them with tenfold cross-validation for 10 iterations (n = 7144) and external validation (n = 2988) and compared them with a standard method to identify patient conditions, the Systematized Nomenclature of Medicine. The gold standard for GIB diagnosis was the independent dual manual review of medical records. The primary outcome was the positive predictive value.A decision rule using GIB-specific terms from ED triage and ED review-of-systems assessment performed better than the Systematized Nomenclature of Medicine on internal validation and external validation (positive predictive value = 85% confidence interval:83%-87% vs 69% confidence interval:66%-72%; P < 0.001). The syntax-based NLP algorithm and Bidirectional Encoder Representation from Transformers neural network-based NLP algorithm had similar performance to the structured-data fields decision rule.An automated decision rule employing GIB-specific triage and review-of-systems terms can be used to trigger EHR-based deployment of risk stratification models to guide clinical decision making in real time for patients with acute GIB presenting to the ED.