Why Use a One-Standard-Error Rule?
Cross-validation can rank regression candidates by estimated prediction error, but the smallest observed error does not necessarily identify the most useful model. Small differences may reflect how the data happened to divide into folds rather than a meaningful predictive advantage. The one-standard-error rule gives a practical way to trade a small amount of estimated predictive accuracy for a simpler model.
The rule starts with the candidate whose mean cross-validation error is lowest. It then allows a model whose error is up to one estimated standard error above that minimum. Among the eligible candidates, select the simplest according to a stated measure, often the number of predictors. This is a selection heuristic; it is not a test proving that the candidates have equal predictive performance.
Calculating the Error and Its Standard Error
For the usual fold-based calculation, first find the sample standard deviation of the \(K\) errors for a candidate:
Here, \(s_m\) describes how much that candidate’s fold errors vary, while \(\operatorname{SE}_m\) estimates the uncertainty in their mean. These are not interchangeable: the one-standard-error cutoff uses an estimated standard error, not the standard deviation of fold errors.
This calculation is a conventional approximation. Fold errors can be dependent because the training sets overlap, so \(\operatorname{SE}_m=s_m/\sqrt K\) should not be interpreted as an exact measure of uncertainty. Use the error and standard-error estimates supplied by a problem when it specifies a method. If calculating from fold errors, use the same folds and metric across candidates where possible, and make clear which estimate is used.
Calculate \(\bar e_m\) for each candidate and identify \(m_{\min}\).
Use \(\operatorname{SE}_{m_{\min}}\), not the standard error of a different candidate.
Add the minimum mean error and its standard error: \(C=\bar e_{m_{\min}}+\operatorname{SE}_{m_{\min}}\).
Include every candidate with mean error at or below \(C\), then compare their complexity.
Why the Standard Error Has This Form
The calculation follows from the variance of an average. If fold errors are treated as independent observations with common variance \(\sigma^2\), then the variance of their mean is:
Taking the square root gives the standard deviation of the mean, \(\sigma/\sqrt K\). Since \(\sigma\) is unknown, the fold sample standard deviation \(s_m\) estimates it, giving \(s_m/\sqrt K\). Cross-validation folds do not perfectly satisfy the independence assumption, which is why this is a practical approximation rather than a guarantee of exact sampling uncertainty.
Worked Example: Choosing Between Three Predictor Sets
A hypothetical pricing study compares three regression candidates using the same six folds. The reported errors are fold-level mean squared errors in squared response units. Candidate A uses six predictors, candidate B uses three, and candidate C uses one. Apply the one-standard-error rule. No interest, mortality, or probability model is assumed; the values are hypothetical cross-validation results.
Candidate A. Its fold errors are \(8,9,10,10,11,12\). Their sum is \(60\), so \(\bar e_A=60/6=10\). Deviations from 10 are \(-2,-1,0,0,1,2\); the squared deviations sum to \(10\). Thus \(s_A=\sqrt{10/5}=\sqrt 2\approx1.4142\), and \(\operatorname{SE}_A=1.4142/\sqrt 6\approx0.5774\).
Candidate B. Its errors are \(9,10,10,11,11,11\), which sum to \(62\), giving \(\bar e_B=62/6\approx10.3333\). The squared deviations from \(10.3333\) sum to approximately \(3.3333\). Therefore \(s_B=\sqrt{3.3333/5}\approx0.8165\), and \(\operatorname{SE}_B\approx0.8165/\sqrt6\approx0.3333\).
Candidate C. Its errors are \(10,11,11,12,12,12\), which sum to \(68\), giving \(\bar e_C=68/6\approx11.3333\). The squared deviations sum to approximately \(3.3333\), so \(s_C\approx0.8165\) and \(\operatorname{SE}_C\approx0.3333\).
Candidate A has the minimum mean error, \(10\). Its one-standard-error cutoff is \(10+0.5774=10.5774\). Candidate B’s mean error, \(10.3333\), is below the cutoff; candidate C’s, \(11.3333\), is above it. A and B are eligible, and B is simpler, so the rule selects the three-predictor candidate. The method avoids selecting A solely because it has the smallest observed mean error.
Check: \(10.3333\leq10.5774\), but \(11.3333>10.5774\); the eligibility decisions match the stated selection.
Worked Example: A Candidate Just Inside the Cutoff
A hypothetical forecasting analysis compares regression models using five common folds. The errors are on the same scale for every candidate. A five-predictor model has errors \(4,5,5,5,6\); a two-predictor model has errors \(4.5,5,5,6,6\); and a one-predictor model has errors \(5,5,6,6,6\). Use the one-standard-error rule and retain enough precision to assess the boundary.
For the five-predictor model, the mean is \((4+5+5+5+6)/5=25/5=5\). Its deviations are \(-1,0,0,0,1\), whose squared values sum to \(2\). Hence \(s=\sqrt{2/4}=\sqrt{0.5}\approx0.7071\), and its standard error is \(0.7071/\sqrt5\approx0.3162\). This model has the minimum mean error, so the cutoff is \(5+0.3162=5.3162\).
The two-predictor model’s mean is \((4.5+5+5+6+6)/5=26.5/5=5.3\). Since \(5.3\leq5.3162\), it is eligible. The one-predictor model’s mean is \((5+5+6+6+6)/5=28/5=5.6\), which exceeds the cutoff. Therefore, the two-predictor model is the simplest eligible candidate and is selected.
Check: The two-predictor model is only \(0.0162\) below the cutoff, but it qualifies because eligibility uses \(\leq\). Rounding the cutoff too early could obscure this boundary comparison.
Worked Example: Using the Wrong Standard Error
A hypothetical regression comparison has five folds. The minimum-error, four-predictor candidate has errors \(4,6,5,5,5\). A simpler, one-predictor candidate has errors \(3,5,5,7,7\). Apply the rule and show why using the simpler candidate’s own standard error to set the cutoff gives the wrong eligibility decision.
The four-predictor candidate has mean error \((4+6+5+5+5)/5=25/5=5\). Its deviations are \(-1,1,0,0,0\), so the squared deviations sum to \(2\). Its sample standard deviation is \(\sqrt{2/4}\approx0.7071\), and its standard error is \(0.7071/\sqrt5\approx0.3162\). The correct cutoff is therefore \(5+0.3162=5.3162\).
The simpler candidate’s mean is \((3+5+5+7+7)/5=27/5=5.4\). Since \(5.4>5.3162\), it is not eligible; with only these two candidates, the four-predictor model is selected.
Common mistake and correction. The simpler candidate’s deviations from \(5.4\) are \(-2.4,-0.4,-0.4,1.6,1.6\), with squared deviations summing to \(11.2\). Its standard error is \(\sqrt{11.2/4}/\sqrt5\approx0.7483\). Incorrectly adding that value to the minimum error gives \(5.7483\), which would make the simpler candidate appear eligible. The cutoff must instead use the standard error belonging to the minimum-error candidate.
Check: The correct comparison is \(5.4>5.3162\), so the one-predictor candidate is outside the eligible set.
Interpreting the Rule Carefully
The rule is designed to favor parsimony when the estimated predictive advantage of a more complicated candidate is modest relative to the uncertainty in the minimum error. The “simplest” candidate should be defined before making the choice. It may mean fewer predictors, or another stated complexity measure if the candidates differ in a more complicated way. If equally simple candidates remain, use a stated tie-breaker, such as the lower mean cross-validation error.
Being within one standard error does not establish that two models have the same true prediction error. Nor does the cutoff serve as a confidence interval or a formal hypothesis test. It is a practical selection rule. The chosen candidate should still be assessed in the context of the modeling objective, the error metric, and the data used to produce the cross-validation estimates.
A few checks prevent common errors:
- Compare mean errors calculated with the same metric and response scale. A lower value has meaning only relative to the metric being used.
- Use the standard error attached to the candidate with the minimum mean error when forming the cutoff.
- Do not substitute the fold-error standard deviation for its standard error.
- Use the unrounded values for eligibility decisions when a candidate is close to the cutoff.
- Apply the simplicity criterion only after finding the entire eligible set; do not automatically choose the model with the very lowest error.
In particular, a small difference between candidate means is not by itself enough to invoke the rule. First calculate the minimum model’s standard error, then compare each candidate with the resulting cutoff, and only then select by complexity. This ordered procedure makes the recommendation reproducible and aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, calculate or use the stated cutoff before choosing a candidate. Treat the listed errors as hypothetical results on a common metric.
- A four-predictor model has the minimum mean cross-validation error of \(8.0\), with estimated standard error \(0.5\). Simpler candidates have mean errors \(8.4\) with two predictors, \(8.5\) with one predictor, and \(8.6\) with no predictors. Which candidate does the rule select? (Answer: The one-predictor candidate, because the cutoff is \(8.5\) and it is the simplest eligible model.)
- The minimum-error candidate has fold errors \(6,8,7,7\). Calculate its mean, sample standard deviation, standard error, and cutoff. A simpler candidate has mean error \(7.3\), and the next simplest has mean error \(7.5\). Which simpler candidate qualifies? (Answer: Mean \(7\), sample standard deviation \(\sqrt{2/3}\approx0.8165\), standard error \(\approx0.4082\), cutoff \(\approx7.4082\); only the candidate with mean \(7.3\) qualifies.)
- The minimum-error candidate has five-fold errors \(2,4,3,3,3\). A simpler candidate has mean error \(3.4\). Does the simpler candidate qualify? (Answer: The minimum mean is \(3\), its standard error is \(\sqrt{2/4}/\sqrt5\approx0.3162\), and the cutoff is \(3.3162\); no, the simpler candidate does not qualify.)
- The minimum mean error is \(12\), and its standard error is \(0.6\). Candidate A has mean error \(12.7\) and is simpler than candidate B, whose mean error is \(12.4\). Which candidate is selected by the rule? (Answer: Candidate B; the cutoff is \(12.6\), so A is ineligible and B is eligible.)
- A candidate with minimum mean error \(5.0\) has standard error \(0.2\). A simpler model has mean error \(5.1\). Does eligibility prove that the simpler model has the same true prediction error as the minimum-error candidate? (Answer: It is eligible because \(5.1\leq5.2\), but eligibility does not prove equal true prediction error; the one-standard-error rule is a selection heuristic.)