Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions

The artificial intelligence chatbot ChatGPT was used to answer questions from final exams administered in two general chemistry courses, including questions with closed-response format and with open-response format. For closed-response questions, ChatGPT was very capable at identifying the concept even when the question included a great deal of chemical symbolism. However, ChatGPT's success at solving problems was only 44%, a value well below the class average of 69%. On open-response questions, ChatGPT's responses displayed strong language processing ability with higher performance on questions that could be solved with more generalizable information compared to questions that required specific skills, especially when those topics or skills were primarily found in lecture. Incorrect responses and flawed explanations were often logically sound and would be persuasive to a novice. The chatbot is currently ill-equipped to provide reliable answers or explanations to students for many representative exam questions, but a potential use is to create assignments in which students analyze and improve ChatGPT's responses.