Calibration set reduction by the selection of a subset containing the best fitting samples showing optimally predictive ability

Near-infrared (NIR) spectroscopy is a rapid, non-invasive and cost-effective technique, for which sample pre-treatment is often not required. It is applied for both qualitative and quantitative analyses in various application fields. Often, large calibration sets are used, from which informative subsets can be selected without a loss of meaningful information. In this study, a new approach for sample subset selection is proposed and evaluated. The global PLS model, obtained with the original large global calibration set after FCAM-SIG variable selection, is used for the selection of the best fitting subset of calibration samples with optimally predictive ability. This best fitting calibration subset is called the optimally predictive calibration subset (OPCS). After ranking the global calibration samples according to increasing residuals, different enlarging fractions of the ranked calibration set are selected. For each fraction, the optimal predictive ability and the corresponding optimal PLS complexity are determined by cross model validation (CMV). After performing CMV with all fractions, the fraction with the best fitting samples and optimally predictive ability, i.e. the OPCS, is determined. The use of the best fitting samples from the global PLS model results in an OPCS-based model which is similar to the global PLS model and has a similar predictive ability. Because the best fitting samples do not need to be representative for the global calibration set, but only need to support the OPCS-based model, the number of samples in the OPCS model is mostly smaller than that selected by a traditional representative sample subset selection method. The new OPCS approach is tested on three real life NIR data sets with twelve X-y combinations to model. The results show that the number of selected samples obtained by the OPCS approach is statistically significantly lower than (i) that of the most suitable and widely used representative sample selection method of Kennard and Stone, and (ii) that suggested by the guideline that the optimal sample size N for reduced calibration sets should surpass the PLS model complexity A by a factor 12. An additional advantage of the OPCS approach is that no outliers are included in the subset because only the best fitting calibration samples are selected. In the new OPCS approach, two additional innovations are built in: (i) CMV is for the first time applied for sample selection and (ii) in CMV, the "one standard error rule", adopted from "Repeated Double Cross Validation", is for the first time used for the determination of the optimal PLS complexity of the OPCS-based models.