Developing and evaluating an empirically-based diagnostic checklist for assessing second language integrated writing

Writing in academic contexts typically includes integrated or source-based writing tasks. Given the significance of identifying students’ strengths and weaknesses and providing them with fine-grained feedback, this study aimed at developing and validating an empirically-based, binary-choice diagnostic checklist for L2 reading-listening-writing (RLW) integrated tasks. First, with reference to L2 students’ verbalizations of their challenges in integrated writing tasks, a diagnostic checklist was developed and then examined and revised by two ESL writing experts, resulting in the construction of a checklist comprising 30 items. Each item was a one-sentence descriptor that addressed a specific aspect of an integrated essay. The study evaluated the checklist’s function in providing reliable and effective diagnostic information by examining score consistency across raters on different essay prompts, raters’ confidence in rating, relationship between the checklist scores and scores awarded using TOEFL’s rating scale, and students’ perception of the feedback provided through the checklist. The results supported the benefits of using a descriptor-based checklist for assessing students’ performance in integrated writing and providing fine-grained diagnostic feedback. Implications of the use of the checklist to provide diagnostic feedback to students and to apply cognitive diagnostic models to writing data are also discussed.