Examining the Scoring of Content Integration in a Listening-Speaking Test: A G-Theory Analysis

Communication in the real world often entails the interpretation, evaluation, and integration of content from different sources. However, it appears that the ability to integrate content into discourse has not been explicitly scored for in existing studies. This study operationalizes content integration in the analytic scoring of a listening-speaking test in Chinese. International students who were non-native speakers of Chinese took the test that comprised two retell tasks and an oral presentation linked by an academic scenario. They were scored for content integration, organization, delivery, and language control for all three tasks. Multivariate generalizability theory (G-theory) was used to investigate the functioning of content integration in the analytic rubric of the retell tasks and oral presentation respectively. Overall, this study aimed to illuminate issues on dependability and construct validity for the two task types, focusing on content integration particularly. The findings suggested that content integration functioned differently to some extent when compared with the other dimensions studied.