Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy

Abstract

Objective

People with epilepsy desire to acquire accurate information about epilepsy and actively engage in its management throughout the long journey of living with seizures. ChatGPT is a large language model and we aimed to assess the accuracy and consistency of ChatGPT in responding to the common concerns of people with epilepsy and to evaluate its ability to provide emotional support.

Methods

Questions were collected from the International League against Epilepsy and the China Association against Epilepsy. The responses were independently assessed by two board-certified epileptologists from the China Association against Epilepsy, and a third reviewer resolved disagreements. The reviewers assessed its ability to provide emotional support subjectively.

Results

A total of 378 questions related to epilepsy and 5 questions related to emotional support were included. ChatGPT provided "correct and comprehensive" answers to 68.4% of the questions. The model provided reproducible answers for 82.3% questions. The model performed poorly in answering prognostic questions, with only 46.8% of the answers rated as comprehensive. When faced with questions requiring emotional support, the model can generate natural and understandable responses.

Significance

ChatGPT provides accurate and reliable answers to patients with epilepsy and is a valuable source of information. It also provides partial emotional support, potentially assisting those experiencing emotional distress. However, ChatGPT may provide incorrect responses, leading users to inadvertently accept incorrect and potentially dangerous advice. Therefore, the direct use of ChatGPT for medical guidance is not recommended and its primary use at present is in patients education.