Using Natural Language Processing (NLP) Methods to Predict Topics Included in 2019 Ohio Syphilis Disease Intervention Specialist (DIS) Records

Abstract Background Free-text notes in disease intervention specialist (DIS) records may contain relevant information for STI control. In their current form, the notes are not analyzable without manual reading, which is labor-intensive and prone to error. Methods We used natural language processing (NLP) methods to analyze 2019 Ohio DIS syphilis records with non-missing notes (n = 1,987). We identified 21 topics relevant for transmission and case investigations. We manually coded these records to create “gold standard” labels for each topic (0 = topic not present, 1 = topic present), then trained machine learning models to identify the topics in the text. For models to analyze text data, the text must be converted to numbers. We explored two approaches to numerically represent words: (1) term frequency, inverse document frequency (TF-IDF), which measures importance of words based on how many times they appear in a record and in the dataset as a whole, and (2) GloVe embeddings, which are numerical vectors that were developed by researchers for each word in the English language to encode its semantic meaning. We explored three types of statistical models (naïve Bayes, support vector machine [SVM], and logistic regression) using TF-IDF, and one type of neural network model (long short-term memory [LSTM] model) using GloVe. All models were used for binary prediction (i.e., topic not present, topic present). Results For most topics, the LSTM model performed the best overall in identifying topics, and the SVM model performed the best among the statistical models. For example, the LSTM model predicted the topic “substance use” with high accuracy (97%), sensitivity (92%), and specificity (98%). No model performed well for uncommon topics (e.g., “alcohol use” or “delays in care”). Conclusions Machine learning models performed well in identifying some topics in 2019 Ohio syphilis records. This analysis is a first step in applying NLP methods to making DIS notes more accessible for analysis.