The reliability of categorical triple collocation for evaluating soil freeze/thaw datasets

Seasonal soil freeze/thaw (FT) state transition plays a critical role in the range of ecosystem, hydrological and biogeochemical processes. A thorough and large-scale validation of remote-sensed or model-based FT products is therefore quite important. Previous validation studies have applied categorical triple collocation (CTC) as a cross-validation method to estimate the relative performance ranking of various FT datasets, including in situ observations. While CTC has proven useful for qualitatively evaluating FT datasets, quantitative estimates of classification accuracy, which has not yet been assessed against direct validation results, would be even more valuable. To fill this gap, we compare CTC estimated performance rankings and quantitative classification accuracies with those obtained from dense soil temperature and sparse surface temperature observations from April 2015 through December 2019. CTC estimated classification accuracies are found to be strongly correlated (r > 0.927) with dense ground observations, along with very low bias (< 0.038) and RMSE (< 0.086). However, the bias and RMSE of CTC-estimated freeze accuracies are significantly inflated when sparse surface temperatures are used instead as the benchmark. Small errors are found with low absolute values (<0.317) of CTC-estimated class imbalance and a sample size of at least 365. CTC can generally provide the correct performance ranking for each product within a triplet - with low risk of incorrectly ranking all three products. In addition, a sample size of 10–160 is adequate for CTC to provide the correct ranking for the highest- or lowest-ranked product. This improves our knowledge and understanding of the reliability of the CTC method.