Chi-square Difference Tests for Comparing Nested Models: An Evaluation with Non-normal Data

The relative fit of two nested models can be evaluated using a chi-square difference statistic. We evaluate the performance of five robust chi-square difference statistics in the context of confirmatory factor analysis with non-normal continuous outcomes. The mean and variance corrected difference statistics performed adequately across all conditions investigated. In contrast, the mean corrected difference statistics required larger samples for the p-values to be accurate. Sample size requirements for the mean corrected difference statistics increase as the degrees of freedom for difference testing increase. We recommend that the mean and variance corrected difference testing be used whenever possible. When performing mean corrected difference testing, we recommend that the expected information matrix is used (i.e., choice MLM), as the use of the observed information matrix (i.e., choice MLR) requires larger samples for p-values to be accurate. Supplementary materials for applied researchers to implement difference testing in their own research are provided.