Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale

As artificial intelligence (AI) became a part of daily life, it has become important to determine user competence in using AI technology. Here, we propose the concept of AI literacy and develop a quantitative scale for obtaining accurate data regarding the AI literacy of ordinary users. We first identified the primary core constructs of AI literacy, including awareness, use, evaluation, and ethics. Next, we generated 65 items to capture these four constructs; only 31 items were retained after a three-step content validation process. Then, we conducted a survey, and collected two samples of data. By reducing the number of items using the first sample and performing reliability and validity tests on the second sample, we obtained a 12-item instrument for the quantitative measurement of AI literacy. The results confirmed that the proposed four-construct model is an adequate representation of AI literacy. Further, AI literacy is significantly related to digital literacy, attitude towards robots, and users’ daily usage of AI. This study will not only aid researchers in understanding how user competence in using AI technology affects human–AI interactions but will also help designers develop AI applications that are aligned with the AI literacy levels of the target users.