The One-Standard-Error Rule and Its Limits

What the One-Standard-Error Rule Does

Cross-validation often produces a small set of candidate models with similar estimated risks. Choosing the candidate with the smallest estimated risk is one option, but that minimum can be sensitive to estimation noise. The one-standard-error rule responds by preferring a simpler candidate whose estimated risk is close to the minimum. Its purpose is a selection preference, not a proof that the selected model has the lowest risk.

Let the candidates be indexed by $j=1,\ldots,J$. Write $\widehat R_j$ for the cross-validation risk estimate of candidate $j$, and let $c_j$ denote its complexity. Assume for now that the candidates have an unambiguous simplicity ordering: smaller $c_j$ means simpler. Define $j_{\min}$ to be a candidate with the smallest estimated risk. The usual rule computes a standard error $s_{\min}$ for that candidate’s cross-validation estimate, then selects the simplest candidate satisfying the threshold.

Definition: Given cross-validation estimates $\widehat R_1,\ldots,\widehat R_J$, a minimizing candidate $j_{\min}\in\arg\min_j\widehat R_j$, and a chosen standard-error estimate $s_{\min}$ for $\widehat R_{j_{\min}}$, the one-standard-error rule selects a simplest candidate among $$ \left\{j:\widehat R_j\leq \widehat R_{j_{\min}}+s_{\min}\right\}. $$ The rule requires a stated simplicity ordering. If candidates are incomparable in complexity, “simplest” needs an additional decision convention.

In $K$-fold cross-validation, a commonly used fold-based estimate is the sample standard deviation of the $K$ fold scores divided by $\sqrt K$. This calculation treats the fold scores as if they supplied $K$ independent observations of a common quantity. That interpretation is not generally justified: the training sets overlap, and fold scores can be dependent. We will use the calculation in examples while keeping its limitations explicit.

Calculating the Rule

Worked Example: Choosing Among Three Model Sizes

Suppose three candidates have complexities $c=1,3,10$, where smaller complexity means simpler. Their cross-validation risk estimates are $0.214,0.200,0.193$, respectively. The minimum is $0.193$, attained by the candidate with $c=10$. Suppose its reported standard error is $0.018$.

The one-standard-error threshold is $$ 0.193+0.018=0.211. $$ The candidate with $c=1$ has estimate $0.214$, which exceeds the threshold. The candidates with $c=3$ and $c=10$ are within it, since $0.200\leq0.211$ and $0.193\leq0.211$. The rule therefore selects $c=3$, the simplest eligible candidate.

This choice does not claim that the $c=3$ model has smaller true risk than the $c=10$ model. It deliberately accepts a somewhat larger estimated risk in exchange for lower complexity.

Worked Example: Computing a Fold-Based Standard Error

For one candidate, suppose the five fold scores are $0.18,0.22,0.20,0.24,0.16$. Their mean is $$ \bar R_{\mathrm{fold}}=\frac{0.18+0.22+0.20+0.24+0.16}{5}=0.20. $$

The deviations from the mean are $-0.02,0.02,0,0.04,-0.04$. The sum of their squared deviations is $$ 0.0004+0.0004+0+0.0016+0.0016=0.004. $$ Thus the sample variance is $0.004/(5-1)=0.001$, the sample standard deviation is $\sqrt{0.001}\approx0.03162$, and the usual fold-based standard error calculation gives $$ \frac{0.03162}{\sqrt{5}}\approx0.01414. $$ If this candidate has the smallest mean cross-validation risk, the corresponding one-standard-error threshold is approximately $0.20+0.01414=0.21414$. A simpler candidate with estimated risk $0.211$ would be eligible; one with estimated risk $0.216$ would not.

The arithmetic defines the reported quantity. Whether $0.01414$ accurately estimates the sampling standard deviation of the cross-validation mean is a separate question.

Why the Fold Scores Do Not Usually Give a Risk Certificate

The fold-based calculation is sometimes read as if it established that any candidate within one reported standard error of the minimum has essentially the same risk, or that the selected candidate is risk-optimal. Neither conclusion follows from the rule. First, a standard error is a measure of sampling variability only when its assumptions and target are appropriate. Second, even a valid standard error for each fixed candidate does not, by itself, provide simultaneous control across all candidates searched and then selected.

To see the dependence issue, write $Z_k$ for the risk score on fold $k$, and suppose the reported cross-validation mean is $\bar Z=K^{-1}\sum_{k=1}^K Z_k$. If the fold scores have common variance $\sigma^2$, then direct expansion gives

\operatorname{Var}(\bar Z) =\frac{1}{K^2}\left(K\sigma^2+2\sum_{k<\ell}\operatorname{Cov}(Z_k,Z_\ell)\right).

The familiar independent-score formula $\sigma^2/K$ follows only when all pairwise covariances are zero. In cross-validation, the validation portions are disjoint, but the fitted models use heavily overlapping training data; the resulting scores need not be independent. Disjoint validation sets alone do not establish zero covariance.

Result: If the $K$ fold scores have common variance $\sigma^2$ and common pairwise correlation $\rho$, then $$ \operatorname{Var}(\bar Z)=\frac{\sigma^2}{K}\bigl(1+(K-1)\rho\bigr). $$ Proof: For each distinct pair, $\operatorname{Cov}(Z_k,Z_\ell)=\rho\sigma^2$. There are $K(K-1)/2$ such pairs, so the variance expansion yields $$ \operatorname{Var}(\bar Z) =\frac{1}{K^2}\left(K\sigma^2+K(K-1)\rho\sigma^2\right) =\frac{\sigma^2}{K}\bigl(1+(K-1)\rho\bigr). $$ Thus the independence formula omits the covariance contribution unless $\rho=0$. $\square$

Worked Example: A Dependence Effect on the Standard Error

Suppose $K=5$ fold scores have variance $\sigma^2=0.01$ and common correlation $\rho=0.75$. The variance of their mean is $$ \frac{0.01}{5}\bigl(1+4(0.75)\bigr) =0.002(4)=0.008, $$ so its standard deviation is $\sqrt{0.008}\approx0.0894$. If one incorrectly used the independent-score variance $0.01/5=0.002$, the reported standard deviation would be $\sqrt{0.002}\approx0.0447$, half as large.

This calculation isolates the effect of positive dependence. It does not assert that every cross-validation design has correlation $0.75$; it shows why the fold-score spread alone cannot generally certify the uncertainty of the average. In the extreme case $Z_1=\cdots=Z_K$, the sample standard deviation across folds is zero even though the common score can still vary across repeated data sets. Identical fold scores do not imply a nonrandom estimate.

A Risk Bound Requires More Than the Rule

A simple deterministic comparison makes clear what would be needed for a risk guarantee. Let $R_j$ be the risk of candidate $j$ for a specified target, such as the risk of the procedure trained at a fixed sample size. Suppose the selected candidate $\hat j$ satisfies $\widehat R_{\hat j}\leq\min_j\widehat R_j+t$, for some nonnegative threshold $t$. If every estimate is uniformly accurate to within $\varepsilon$, then the selected candidate’s risk cannot be too much larger than the best candidate risk.

Result: If $\max_j|\widehat R_j-R_j|\leq\varepsilon$ and $\widehat R_{\hat j}\leq\min_j\widehat R_j+t$, then $$ R_{\hat j}\leq\min_j R_j+t+2\varepsilon. $$ Proof: Uniform accuracy gives $R_{\hat j}\leq\widehat R_{\hat j}+\varepsilon$. By the selection condition, $$ R_{\hat j}\leq\min_j\widehat R_j+t+\varepsilon. $$ For every $j$, uniform accuracy also gives $\widehat R_j\leq R_j+\varepsilon$; hence $\min_j\widehat R_j\leq\min_j R_j+\varepsilon$. Combining the inequalities proves the claim. $\square$

This result is a conditional comparison, not a guarantee supplied by the one-standard-error rule. It requires a bound that holds simultaneously over the candidate set, and it requires the estimates and risks to refer to a compatible target. Setting $t=s_{\min}$ is useful only if the threshold is suitably controlled; a fold-based standard error is not itself a uniform error bound. Moreover, the one-standard-error rule selects by simplicity within a risk band, rather than minimizing estimated risk exactly.

Other Limits on Interpretation

The risk target also matters. In $K$-fold cross-validation, each fitted candidate uses roughly $n(K-1)/K$ observations, while the final model is often refit on all $n$ observations. If the target is the risk of that full-data refit, the cross-validation estimate may target performance at a smaller training size instead. The difference can matter for learning procedures whose risk changes appreciably with sample size. A small fold-based standard error does not remove this target mismatch.

Selection adds another complication. The candidate with the smallest observed risk was chosen after comparing several noisy estimates. Its minimum is therefore not an ordinary estimate for a candidate fixed in advance. The one-standard-error rule changes which candidate is reported; it does not undo the data-dependent search or automatically quantify uncertainty after selection. Those are distinct inferential questions.

Worked Example: A Simple Model Can Still Be Worse

Suppose the true target risks of a simple and a complex candidate are $0.30$ and $0.20$, respectively. In one data set, their cross-validation estimates are $0.24$ and $0.23$. If the complex candidate is the estimated-risk minimum and its reported standard error is $0.02$, the threshold is $0.25$. The simple candidate qualifies because $0.24\leq0.25$, so the rule selects it.

The chosen model is simpler, as intended, but its true risk in this example is $0.30$, not $0.20$. The estimated difference is small relative to the stated threshold, yet the true difference is $0.10$. This outcome is possible when the estimates are inaccurate or the threshold is poorly calibrated. It demonstrates why “within one standard error” is not equivalent to “the true risks are equal” or “the simpler candidate is no worse.”

The rule is most defensible as a transparent operational preference when the candidate set, complexity ordering, validation design, and reported standard error are all stated. It can be useful when a modest estimated-risk increase is acceptable in return for a simpler or more stable model. It should not be presented as a confidence interval, a hypothesis test, or a certificate of oracle risk.

Common Pitfall

Wrong reasoning: “The selected model lies within one standard error of the minimum, so it is statistically indistinguishable from the minimum-risk model and is guaranteed to perform nearly as well.”
Correction: The rule only compares observed cross-validation estimates against a threshold. The fold-based standard error may not reflect dependence among fold scores, does not generally account for searching across candidates, and may concern a different training-size target from the final refit. A risk guarantee needs suitable error control for the relevant candidates and target.

When reporting the rule, state how the standard error was calculated, which candidate supplied it, what complexity ordering determined “simplest,” and whether the risk estimates target the final training procedure. These details make the preference reproducible without overstating what it establishes.

Check Your Understanding

Answer each question by distinguishing the calculation from the guarantee it does or does not provide.

Three candidates have cross-validation risks $0.31,0.28,0.27$ in increasing order of complexity. The minimum-risk candidate’s reported standard error is $0.03$. Which candidates satisfy the one-standard-error threshold, and which one is selected?
Why does having disjoint validation folds not, by itself, imply that fold scores are independent?
Under the common-correlation model, what happens to the variance of the fold-score average when $\rho>0$, compared with the independence formula?
State the uniform accuracy condition used in the risk comparison result. Why is a standard error for only the minimizing candidate not enough to establish it?
Give one reason that cross-validation risk estimates may not match the risk target of the final refitted model.