On the Use of Content Specialists in the Assessment of Criterion-Referenced Test Item Validity.

Essential for an effective criterion-referenced testing program is a sei of test items that are valid indicators of the objectives they have been designed to measure. Unfortunately, the complex matter of assessing item validity has received only limited attention from educational measurement specialists. One promising approach to the item validity question is through the collection and analysis of the judgements of content specialists. The purposes of this paper are twofold: First, we will discuss several possible rating forms and statistical methods for the analysis of content specialists' data. Second, we will present the results of our item validation work with science teachers and three of the more promising rating forms. The overall results of the study clearly support the recommendation for expanded use of content specialists' ratings in the item validation process. 2 U 5. DEPARWAENTOF NEALTN, EDUCATION W Hambleton & Novick, 1973; Livingston, 1972; Millman, 1974; and Popham & Nusek, 1969). Considering its importance, educational measurement specialists have given relatively little attention to the problem of item validation, i.e., the problem concerning the extent to which items are measures of tL, jectives they have been designed to measure. The problem of item validation is of particular importance with criterion-referenced tests because of the way the test score information is used. The success of an individualized program depends to a considerable 1 A paper presented at the annual meeting of AERA, San Francisco, 1976. The paper has been published as Laboratory of Psychometric and Evaluative Research Report No.24. Amherst, Mass: The University of Massachusetts, 1976.