AI‐Assisted Assessment of Inquiry Skills in Socioscientific Issue Contexts

ABSTRACT Background Study Assessing learners' inquiry‐based skills is challenging as social, political, and technological dimensions must be considered. The advanced development of artificial intelligence (AI) makes it possible to address these challenges and shape the next generation of science education. Objectives The present study evaluated the SSI inquiry skills of students in an AI‐enabled scoring environment. An AI model for socioscientific issues that can assess students' inquiry skills was developed. Responses to a learning module were collected from 1250 participants, and the open‐ended responses were rated by humans in accordance with a designed rubric. The collected data were then preprocessed and used to train an AI rater that can process natural language. The effects of two hyperparameters, the dropout rate and complexity of the AI neural network, were evaluated. Results and Conclusion The results suggested neither of the two hyperparameters was found to strongly affect the accuracy of the AI rater. In general, the human and AI raters exhibited certain levels of agreement; however, agreement varied among rubric categories. Discrepancies were identified and are discussed both quantitatively and qualitatively.