AUTOMATED MACHINE LEARNING WITH AUTOGLUON TO PREDICT POSTOPERATIVE PNEUMONIA USING THE AMERICAN COLLEGE OF SURGEONS’ NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM DATABASE

SESSION TITLE: Late Breaking Chest Infections PostersSESSION TYPE: Original Investigation PostersPRESENTED ON: 10/18/2022 01:30 pm - 02:30 pmPURPOSE: Risk calculators to predict post-operative pneumonias often rely on logistic regression (LR) analysis. Major risk calculators, including ARISCAT, are based on logistic regressions that only take into account a limited number of variables. Machine learning models are able incorporate a greater number of input variables by identifying non-linear relationships. Automated machine learning (AutoML) processes regularly outperform regular machine learning (ML) and LR methods for predictive accuracy. AutoML systems have not yet been applied to predict post-operative pneumonia.METHODS: We used an AutoML system developed and released by Amazon in 2020, AutoGluon v0.3.1, to predict post-operative pneumonia in the 2019 ACS NSQIP database. Post-operative pneumonia was defined as a pneumonia that occurred within 30 days of the surgery. Models were trained for four hours to optimize performance on the Brier score, with lower being better. Validation of all performance metrics was done using the 2019 ACS NSQIP database. Each model was plotted on AUROC and AUC-PR curves to compare performance across different models.RESULTS: Our dataset included 3,049,617 unique patients, with a median age of 58.0 and 56.8% female. Our patient set was diverse, with 69.8% of patients being white and 7.9% of Hispanic descent. Of all the patients in the dataset, there were 27,167 post-operative pneumonias (0.9%). Brier scores were calculated for each model with the top performing model being an ensembled LightGBM model having a Brier score of 0.0084 on the validation set. The corresponding AUROC and AUC-PR was 0.879 and 0.072 respectively.CONCLUSIONS: Automated machine learning models offer similar if not better discriminatory characteristics to existing post-operative pneumonia calculators. Benefits of these models include recognition of non-linear relationships between variables and a higher number of variables incorporated into model construction. Our AutoML models, and specifically our top-performing ensembled LightGBM model had Brier scores, AUROC, and AUC-PR that were similar or better than those of currently used logistic regression risk calculators. Based on these results, AutoML analyses should be considered for risk estimation of post-operative pneumonia.CLINICAL IMPLICATIONS: Establishing more accurate, reliable, and holistic risk stratification model has the potential to better assess the need for pre-operative risk optimization and ascertain which patients are optimized from a pulmonary perspective to procede to the operating room. Logistic regressions to date have provided useful but incomplete prediction models for important perioperative outcomes. AutoML affords the opportunity to consider a wide range of demographic, epidemiologic, and clinical factors to enhance our ability to discern perioperative risk and improve our perioperative outcomes.DISCLOSURES:No relevant relationships by Yin AphinyanaphongsNo relevant relationships by Shelly BrejtNo relevant relationships by Kenneth BrillNo relevant relationships by Ji ChenNo relevant relationships by Eric HuNo relevant relationships by Karen KanNo relevant relationships by Gregory KatzNo relevant relationships by Andrew KelleherNo relevant relationships by Jacob MartinNo relevant relationships by Jacob OedingNo relevant relationships by Nicolai OstbergNo relevant relationships by RUDRA RAMANATHANNo relevant relationships by Avinash Thangirala SESSION TITLE: Late Breaking Chest Infections Posters SESSION TYPE: Original Investigation Posters PRESENTED ON: 10/18/2022 01:30 pm - 02:30 pm PURPOSE: Risk calculators to predict post-operative pneumonias often rely on logistic regression (LR) analysis. Major risk calculators, including ARISCAT, are based on logistic regressions that only take into account a limited number of variables. Machine learning models are able incorporate a greater number of input variables by identifying non-linear relationships. Automated machine learning (AutoML) processes regularly outperform regular machine learning (ML) and LR methods for predictive accuracy. AutoML systems have not yet been applied to predict post-operative pneumonia. METHODS: We used an AutoML system developed and released by Amazon in 2020, AutoGluon v0.3.1, to predict post-operative pneumonia in the 2019 ACS NSQIP database. Post-operative pneumonia was defined as a pneumonia that occurred within 30 days of the surgery. Models were trained for four hours to optimize performance on the Brier score, with lower being better. Validation of all performance metrics was done using the 2019 ACS NSQIP database. Each model was plotted on AUROC and AUC-PR curves to compare performance across different models. RESULTS: Our dataset included 3,049,617 unique patients, with a median age of 58.0 and 56.8% female. Our patient set was diverse, with 69.8% of patients being white and 7.9% of Hispanic descent. Of all the patients in the dataset, there were 27,167 post-operative pneumonias (0.9%). Brier scores were calculated for each model with the top performing model being an ensembled LightGBM model having a Brier score of 0.0084 on the validation set. The corresponding AUROC and AUC-PR was 0.879 and 0.072 respectively. CONCLUSIONS: Automated machine learning models offer similar if not better discriminatory characteristics to existing post-operative pneumonia calculators. Benefits of these models include recognition of non-linear relationships between variables and a higher number of variables incorporated into model construction. Our AutoML models, and specifically our top-performing ensembled LightGBM model had Brier scores, AUROC, and AUC-PR that were similar or better than those of currently used logistic regression risk calculators. Based on these results, AutoML analyses should be considered for risk estimation of post-operative pneumonia. CLINICAL IMPLICATIONS: Establishing more accurate, reliable, and holistic risk stratification model has the potential to better assess the need for pre-operative risk optimization and ascertain which patients are optimized from a pulmonary perspective to procede to the operating room. Logistic regressions to date have provided useful but incomplete prediction models for important perioperative outcomes. AutoML affords the opportunity to consider a wide range of demographic, epidemiologic, and clinical factors to enhance our ability to discern perioperative risk and improve our perioperative outcomes. DISCLOSURES: No relevant relationships by Yin Aphinyanaphongs No relevant relationships by Shelly Brejt No relevant relationships by Kenneth Brill No relevant relationships by Ji Chen No relevant relationships by Eric Hu No relevant relationships by Karen Kan No relevant relationships by Gregory Katz No relevant relationships by Andrew Kelleher No relevant relationships by Jacob Martin No relevant relationships by Jacob Oeding No relevant relationships by Nicolai Ostberg No relevant relationships by RUDRA RAMANATHAN No relevant relationships by Avinash Thangirala