ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research

Generative large language models (GLLMs), such as ChatGPT and GPT-4 by OpenAI, are emerging as powerful tools for textual analysis tasks in accounting research. GLLMs can solve any textual analysis task solvable using nongenerative methods as well as tasks previously only solvable using human coding. Whereas GLLMs are new and powerful, they also come with limitations and present new challenges that require care and due diligence. This paper highlights the applications of GLLMs for accounting research and compares them with existing methods. It also provides a framework on how to effectively use GLLMs by addressing key considerations, such as model selection, prompt engineering, and ensuring construct validity. In a case study, I demonstrate the capabilities of GLLMs by detecting nonanswers in earnings conference calls, a traditionally challenging task to automate. The new GPT method achieves an accuracy of 96% and reduces the nonanswer error rate by 70% relative to the existing Gow et al. (2021) method. Finally, I discuss the importance of addressing bias, replicability, and data sharing concerns when using GLLMs. Taken together, this paper provides researchers, reviewers, and editors with the knowledge and tools to effectively use and evaluate GLLMs for academic research. This paper was accepted by Eric So, accounting. Funding: Supported by the Foster School of Business – University of Washington. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.03253 .