ChatGPT accurately performs genetic counseling for gynecologic cancers

Abstract

Objective

Artificial Intelligence (AI) systems such as ChatGPT can take medical examinations and counsel patients regarding medical diagnosis. We aim to quantify the accuracy of the ChatGPT V3.4 in answering commonly asked questions pertaining to genetic testing and counseling for gynecologic cancers.

Methods

Forty questions were formulated in conjunction with gynecologic oncologists and adapted from professional society guidelines and ChatGPT version 3.5 was queried, the version that is readily available to the public. The two categories of questions were genetic counseling guidelines and questions pertaining to specific genetic disorders. The answers were scored by two attending Gynecologic Oncologists according to the following scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect, and 4) completely incorrect. Scoring discrepancies were resolved by additional third reviewer. The proportion of responses earning each score were calculated overall and within each question category.

Results

ChatGPT provided correct and comprehensive answers to 33/40 (82.5%) questions, correct but not comprehensive answers to 6/40 (15%) questions, partially incorrect answers to 1/40 (2.5%) questions, and completely incorrect answers to 0/40 (0%) questions. The genetic counseling category of questions had the highest proportion of answers that were both correct and comprehensive with ChatGPT answering all 20/20 questions with 100% accuracy and were comprehensive in responses. ChatGPT performed equally in the specific genetic disorders category, with 88.2% (15/17) and 66.6% (2/3) correct and comprehensive answers to questions pertaining to hereditary breast and ovarian cancer and Lynch syndrome questions respectively.

Conclusion

ChatGPT accurately answers questions about genetic syndromes, genetic testing, and counseling in majority of the studied questions. These data suggest this powerful tool can be utilized as a patient resource for genetic counseling questions, though more data input from gynecologic oncologists would be needed to educate patients on genetic syndromes.