CORR Insights®: A Tool to Estimate Risk of 30-day Mortality and Complications After Hip Fracture Surgery: Accurate for Some but Not All Purposes? A Study From the ACS-NSQIP Database

Where Are We Now? Hip fractures are common, and most patients with hip fractures treated operatively will have better outcomes than those treated nonoperatively [2]. Beside the well-known mortality benefit, patients treated with surgery are discharged from the hospital earlier, are more likely to have healed fractures without shortening or deformity, and are more likely to return to their preinjury level of independence [4]. But because patients with hip fractures present with a variety of demographic and comorbid characteristics, a patient-individualized risk assessment or decision tool would be useful. Harris et al. [5] used the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) database to create and internally validate a freely available online tool to predict the likelihoods of 30-day postoperative mortality and major complications after operative treatment of hip fractures. Patient age, gender, BMI, and 14 other patient-specific health characteristics were used to compute patient-individualized risks. The authors found an overall incidence of mortality of 5% and reported good-to-fair performance and calibration of their model after tenfold internal cross-validation. The authors concluded this tool could be useful for personalized informed consent and shared decision-making with patients and their families. Where Do We Need To Go? Predictive tools run into a number of typical issues when they seek to determine the likelihood of rare events. For example, if the expected risk of death for all-comers is 5%, then a model that simply predicts all patients will survive will be correct 95% of the time. Such a model looks great on paper, but it would be of little clinical utility. Among patient-individualized risk prediction tools that are available online, the tool of Harris et al. [5] offers some interface and usability advantages compared with the online tool available directly from the ACS (http://www.riskcalculator.facs.org/RiskCalculator/). Although this tool may not enhance all informed consent processes, one could imagine referring especially analytically minded or tech-savvy patients and families to this tool to help them understand the gravity of the injury. As the boundaries of research questions and topics expand, it is important that researchers continue to exercise care in their use of big data. One set of concerns pertains to biases and discrimination associated with race and ethnicity [1, 14]. For example, Harris et al. [5] noted that 79% of patients in their study identified as White, while recent United States Census data found that fewer than 62% of all Americans identified as White [13]. Although the discrepancy is almost certainly incidental, the ACS-NSQIP sample does not necessarily represent the population of the United States in terms of racial or ethnic distribution. Large databases may underrepresent some Americans, and these discrepancies could induce major biases depending on the regions of interest, populations or outcomes studied, and how carefully and critically researchers interpret the data. The most important biases may be related to inherent differences between the hospitals that report data to the ACS-NSQIP database and the remaining majority of hospitals in the United States that do not contribute data. Although Harris et al. [5] used tenfold internal validation techniques, it is unclear how well their model would perform if it were validated using an external database such as the ACS-National Trauma Data Bank or ACS-Trauma Quality Improvement Program, Agency for Healthcare Research Quality’s National Inpatient Sample, the National Hospital Discharge Survey of the Centers for Disease Control, or even a database based on administrative claims data such as PearlDiver, MarketScan, or Centers for Medicare & Medicaid Service’s Medicare standard analytic files. How Do We Get There? Given the concerns about the unpredictable impact of reporting idiosyncrasies in various databases, perhaps a good strategy for external validation would be the use of local institutional data in the region, hospital, or practice of interest. For example, a recent study attempted to externally validate six models that predict 30-day mortality after hip fracture against their own institutional data [8], and the authors found that none of the models they studied would have been effective predicting mortality among patients who underwent hip fracture surgery at their hospital. Perhaps the most useful predictive models for a particular location would be those built using the data from patients in that state, region, institution, or integrated health plan. Moving forward, predictive models should be continuously refined, validated, and compared in order to account for changing trends and treatments [6, 11], epidemiology and demographics [7, 12], and even health pandemics [3, 9]. For example, after increased mortality was recognized in patients who tested positive for COVID-19 and were treated for hip fractures, New York University’s Langone Medical Center added COVID-19 as a predictor to their Score for Trauma Triage in the Geriatric and Middle-Aged [9]. Databases should be curated to represent the patients we treat and the outcomes that are important to us and our patients. The major criticisms that are applicable to any ACS-NSQIP study are that the follow-up period is too short (30 days) and there is no nonoperative comparison group. A lack of a nonoperative control group limits our ability to compare the prognoses of patients treated operatively with those treated nonoperatively [10]. Perhaps in the future, a governmental agency, academic collaborative group, or large integrated healthcare organization could create a trauma registry that longitudinally follows patients treated operatively and nonoperatively. But until that day, the most accurate and useful scoring systems and predictive models may be those created using local data.