OP0158 A NOVEL APPROACH FOR MIXED-METHODS RESEARCH USING LANGUAGE LEARNING MODELS: A REPORT USING PATIENTS’ PERSPECTIVES ON BARRIERS TO HIP AND KNEE REPLACEMENT

Background:

While qualitative research techniques provide valuable insights into patient experiences and perceptions in healthcare, analyzing textual data from interviews can be time consuming and require multiple analysts for investigator triangulation. Employing investigator triangulation in qualitative data analysis minimizes potential biases and limitations that stem from relying solely on a single source or investigator. This approach enhances the credibility and validity of the findings by corroborating information from various perspectives and investigators, thus ensuring a more comprehensive and reliable analysis. The capability of Large Language Model (LLM) technology to identify patterns and themes in textual data offers potential for conducting independent analysis. This study explored a novel approach to investigator triangulation in mixed-methods research (MMR) by employing ChatGPT-4, an LLM, for analyzing data from patient interviews.

Objectives:

The objective of this study is to evaluate the application of LLMs, specifically ChatGPT-4, in mixed-methods research. This study will specifically compare the thematic analysis of interview text and the subsequent survey generation conducted by ChatGPT-4 with that performed by human investigators. The comparison will focus on evaluating the quality of the content and assessing the time taken for these tasks. We seek to determine the potential of LLMs in enhancing mixed-method research in healthcare.

Methods:

We compared the thematic analysis and survey generation between investigators and ChatGPT-4, an AI-powered natural language processing tool that utilizes Generative Pre-trained Transformer (GPT), a type of large language model (LLM), as its core model. We used qualitative data from an existing qualitative study that explored patient perceptions of barriers to hip and knee replacement utilization. Textual data from six semi-structured interviews was analyzed by human investigators and ChatGPT-4 independently. Human investigators used NVivo software to analyze the data and generate the codes and themes that emerged from the data. Once the data analysis was completed and it was determined that thematic saturation had been achieved, we proceeded to utilize ChatGPT-4 to directly generate themes from the interview transcripts. Both the human investigators and ChatGPT-4 independently used the generated themes to construct a survey. We compared and evaluated the investigator-generated themes and survey with those generated by ChatGPT-4.

Results:

ChatGPT-4 generated analogous dominant themes and a comprehensive corresponding survey as human investigators but in significantly less time. (Table 1) Human-led thematic analysis produced 6 themes and 30 survey questions, taking significant time. ChatGPT-4 generated the same themes in under 45 minutes and created 17 survey questions in less than 1 minute. Themes generated by ChatGPT-4 covered the same topics as human-generated themes. However, ChatGPT-4's survey questions were shorter and less specific compared to human-generated questions. Figure 1 mixed-methods flowchart proposes integrating LLMs alongside human investigators as a supplementary tool for the preliminary thematic analysis of qualitative data and survey generation.

Conclusion:

The potential of ChatGPT-4 as a tool for assisting in investigator triangulation was evident in both the qualitative portion of mixed methods research (MMR) by assisting with thematic analysis, and in the quantitative portion of MMR by facilitating survey development. This integration may improve the overall workflow efficiency of MMR. Given the novelty of LLM technology and the limited research on its embedded biases, investigators must consider the ethical and qualitative implications when using LLMs for research purposes.

REFERENCES:

NIL.

Acknowledgements:

NIL.

Disclosure of Interests:

Insa Mannstadt: None declared, Susan Goodman UCB (Paid consultant), Novartis (Research support), Mangala Rajan: None declared, Sarah Young: None declared, Fei Wang: None declared, Iris Navarro-Millan Sobi (Swedish Orphan Biovitrum AB) advisory board honoraria, Bella Mehta Novartis (non labelled educational content), Janssen (advisory board).