An automatized semantic analysis of two large-scale listening tests: A corpus-based study

This study examined the semantic features of the simulated mini-lectures in the listening sections of the International English Language Testing System (IELTS) and the Test of English as a Foreign Language (TOEFL) based on automatized semantic analysis to explore the content validity of the two tests. Two study corpora were utilized, the IELTS corpus with 56 mini-lectures (38,944 words) and the TOEFL corpus with 285 mini-lectures (207,296 words). The reference corpus comprised 59 lectures from the Michigan Corpus of Academic Spoken English (MICASE), totaling 571,354 words. The corpora were submitted to automatized semantic tagging using Wmatrix5. Three comparisons were conducted: IELTS versus TOEFL, IELTS versus MICASE lectures, and TOEFL versus MICASE lectures. The results suggest that IELTS and TOEFL mini-lectures shared 78% and 64% of the same semantic features as MICASE, respectively, supporting their relative content validity. Nevertheless, specific semantic categories, such as politics, war, and intimate and sexual relationships, were notably absent from the test corpora, even though they appeared in the academic lecture corpus. In addition, causal connectors are frequently used in both tests, while the mini-lectures of IELTS listening tests cover fewer academic discourse fields than TOEFL mini-lectures. Implications for content validity are discussed.