Cross-institution natural language processing for reliable clinical association studies: a methodological exploration

What is new? Key Findings• We found no setting in which association estimates based on NLP-extracted exposures were consistently similar to using manual labeled variables across different exposures, health outcomes, NLP methods, missing data handling, and use of common measurement error correction (regression calibration).• The error in association was only weakly correlated with the overall F1-score of the NLP models. What this adds to what was known• Association estimates using NLP-extracted exposures may be unreliable, and should be interpreted with caution. What is the implication and what should change now• There is a need for clarity about when an NLP model could be used sufficiently reliably to assess multivariable associations.