What do university students know about Artificial Intelligence? Development and validation of an AI literacy test

Artificial Intelligence (AI) strongly influences our daily lives and work environments. To deal with the challenges as well as to pick up on the opportunities associated with AI, university students need to acquire a basic understanding of AI (so-called AI literacy). To design effective study programs that foster AI competencies, it is necessary to assess the state of students' AI literacy. While there already are some AI literacy tests available, many instruments focus on specific AI courses, rely primarily on self-assessment, or do not provide detailed psychometric information. This study aims to develop and validate an AI literacy test and to provide initial insights into the current state of AI literacy among German university students. We present a validated multiple-choice test to assess AI literacy in higher education. The results suggest that there is a significant variance in AI literacy among students, with most showing a fundamental understanding of AI. AI literacy is higher among students with a technical study background or prior experience in AI. The results imply that there is a need for effective AI courses for a broader audience of students who use AI in their daily lives and will use AI tools in their future careers. Furthermore, educators should consider the variance of students' prior knowledge when designing AI courses.