Risk Consistency Without Correct Model Recovery

Prediction Quality and Model Identity

A model selector returns an index, but the practical goal is often to predict well. These are different objectives. Two models can yield different predictors with identical risk; alternatively, one can be slightly better than another while the difference is too small for the data to reliably distinguish. In either case, a selected predictor can have risk close to the best available risk even when the selected index does not identify a particular model.

This distinction matters under misspecification. The best candidate model need not be the true data-generating model. It is therefore useful to ask whether selection achieves the best risk available within the candidates, separately from whether it recovers a named index. We make that distinction precise and show what additional condition allows index recovery.

Definition: Let \(Z_1,\ldots,Z_n\) be independent observations from a common distribution \(P\), and let \(\ell(Z,f)\) be a loss function for predictor \(f\). Its risk is \(\mathcal R(f)=E[\ell(Z,f)]\), when this expectation exists. For candidate predictors \(f_1,\ldots,f_M\), write \(R_j=\mathcal R(f_j)\) and \(R_{\min}=\min_{1\leq j\leq M}R_j\). A selected predictor \(f_{\hat j}\) is risk consistent relative to these candidates if \(R_{\hat j}-R_{\min}\xrightarrow{p}0\). Model-index recovery means that the selected index converges in probability to a specified target index.

The risk in this definition is the population risk of the selected predictor, not its observed training loss. The target \(R_{\min}\) is the best risk among the candidates being compared. It is not automatically the risk of a true model or the smallest achievable risk over every possible predictor.

A Risk Bound for Penalized Selection

For a fixed candidate predictor \(f_j\), define its empirical risk by

\widehat R_j=\frac{1}{n}\sum_{i=1}^n \ell(Z_i,f_j).

A structural selector may minimize empirical risk plus a nonnegative penalty. The following deterministic argument shows how uniform accuracy of the empirical risks controls the selected predictor’s risk. It also makes clear why a risk bound does not, by itself, name the selected index.

Theorem: Suppose \(M\) is finite, \(R_j\) and \(\widehat R_j\) are finite for every \(j\), and \(\hat j\) minimizes \(\widehat R_j+\operatorname{pen}_j\) over \(j=1,\ldots,M\), where \(\operatorname{pen}_j\geq 0\). On any event where

\max_{1\leq j\leq M}|\widehat R_j-R_j|\leq\delta,

we have

R_{\hat j}-R_{\min}\leq 2\delta+\operatorname{pen}_{j^*},

where \(j^*\) is any index with \(R_{j^*}=R_{\min}\).

Proof. On the stated event, \(R_{\hat j}\leq \widehat R_{\hat j}+\delta\). The definition of \(\hat j\) gives \(\widehat R_{\hat j}+\operatorname{pen}_{\hat j}\leq \widehat R_{j^*}+\operatorname{pen}_{j^*}\). Since \(\operatorname{pen}_{\hat j}\geq0\), it follows that \(\widehat R_{\hat j}\leq\widehat R_{j^*}+\operatorname{pen}_{j^*}\). Finally, \(\widehat R_{j^*}\leq R_{j^*}+\delta\), so

R_{\hat j} \leq \widehat R_{\hat j}+\delta \leq \widehat R_{j^*}+\operatorname{pen}_{j^*}+\delta \leq R_{\min}+2\delta+\operatorname{pen}_{j^*}.

Subtracting \(R_{\min}\) proves the claim. \(\square\)

If the maximum deviation tends to zero in probability and the penalty of a best-risk comparator tends to zero, this inequality gives risk consistency. It does not require the best-risk index to be unique. For model classes containing fitted predictors rather than one fixed predictor each, the same argument applies when the deviation bound holds simultaneously for the predictors being compared.

Worked Example: Two Models With Exactly Equal Risk

Let \(X\) take values \(-1\) and \(1\), each with probability \(1/2\), and let \(Y=0\) with probability one. Use squared-error loss \(\ell((X,Y),f)=(Y-f(X))^2\). Consider two predictors, \(f_1(x)=x\) and \(f_2(x)=-x\).

For either predictor, \(X^2=1\), so the population risks are

R_1=E[(0-X)^2]=E[X^2]=1,\qquad R_2=E[(0+X)^2]=E[X^2]=1.

For every observed sample, not just on average, each empirical risk is also exactly \(1\): every squared residual is \(1\). A selector that breaks ties in favor of index 1 therefore selects index 1 every time, with risk equal to \(R_{\min}=1\). But risk does not distinguish these predictors, which differ at every possible input. There is no unique risk-minimizing index to recover.

This is not a failure of risk consistency. It is a warning against treating a label as if it were identified by the risk criterion when multiple labels attain the same minimum.

When Does Risk Control the Selected Index?

For a finite candidate set with a unique best index, the key quantity is the risk gap between that index and its closest competitor. If that gap stays positive and empirical risks are uniformly accurate, selection must eventually choose the best index. The next result gives a finite-sample version.

Theorem: Suppose \(R_{j_0}<R_j\) for every \(j\ne j_0\), and define the positive separation gap

\Delta=\min_{j\ne j_0}(R_j-R_{j_0})>0.

If \(\hat j\) minimizes the unpenalized empirical risks and \(\max_j|\widehat R_j-R_j|<\Delta/2\), then \(\hat j=j_0\).

Proof. For every \(j\ne j_0\), the assumed deviation bound implies

\widehat R_j >R_j-\frac{\Delta}{2} \geq R_{j_0}+\frac{\Delta}{2} >\widehat R_{j_0}.

Thus every competing index has strictly larger empirical risk than \(j_0\). The empirical minimizer is \(j_0\), as claimed. \(\square\)

Consequently, if the uniform deviation tends to zero in probability and this fixed positive gap holds, the probability of selecting \(j_0\) tends to one. The gap is essential to this argument. If it shrinks with sample size, increasingly accurate risk estimates may still be insufficient to identify which index is best.

Worked Example: A Shrinking Gap and Persistent Index Error

Let \(Y_1,\ldots,Y_n\) be independent \(N(0,1)\) observations. Use squared-error loss and two constant predictors: \(f_0(y)=0\) and \(f_{1,n}(y)=a_n\), where \(a_n=1/n\). (The predictors ignore their input; the notation emphasizes that the second candidate changes with sample size.)

Because \(E[Y_i]=0\) and \(E[Y_i^2]=1\), their population risks are

R_0=E[Y^2]=1,\qquad R_{1,n}=E[(Y-a_n)^2] =E[Y^2]-2a_nE[Y]+a_n^2 =1+\frac{1}{n^2}.

Thus index 0 is the unique best index at each \(n\), but the risk gap is only \(1/n^2\). The difference between the empirical risks is

\widehat R_{1,n}-\widehat R_0 =\frac{1}{n}\sum_{i=1}^n\big((Y_i-a_n)^2-Y_i^2\big) =a_n^2-2a_n\overline Y,

where \(\overline Y=n^{-1}\sum_iY_i\). The empirical selector chooses index 1 precisely when \(\overline Y>a_n/2=1/(2n)\). Since \(\sqrt n\,\overline Y\sim N(0,1)\), this probability is

P(\hat j=1) =P\left(\sqrt n\,\overline Y>\frac{1}{2\sqrt n}\right) \longrightarrow \frac{1}{2}.

The selected index therefore fails to converge in probability to the unique best index 0. Yet the excess risk is either 0 or \(1/n^2\), so \(0\leq R_{\hat j}-R_0\leq 1/n^2\) for every sample. The selected predictor is risk consistent even though index recovery fails.

Empirical Selection Can Switch Between Tied Models

Exact population ties need not produce exact empirical ties. Sampling noise can make either index look better, even when both have the same population risk. The next example shows that this can happen with genuinely different predictions and a conventional empirical-risk selector.

Worked Example: Random Index Selection With Equal Risks

Let \(Y_1,\ldots,Y_n\) be independent Bernoulli random variables with \(P(Y_i=1)=P(Y_i=0)=1/2\). Use squared-error loss and the constant predictors \(g_0=0\) and \(g_1=1\). Their population risks are

R_0=E[Y^2]=\frac12,\qquad R_1=E[(Y-1)^2]=\frac12.

Their empirical-risk difference is

\widehat R_1-\widehat R_0 =\frac{1}{n}\sum_{i=1}^n\big((Y_i-1)^2-Y_i^2\big) =1-2\overline Y.

An empirical minimizer selects \(g_1\) when \(\overline Y>1/2\), selects \(g_0\) when \(\overline Y<1/2\), and encounters a tie when \(\overline Y=1/2\). By symmetry of the binomial distribution, the probabilities of the two strict inequalities are equal. Moreover, the central binomial probability \(P(\overline Y=1/2)\) tends to zero (and is zero for odd \(n\)). Hence each strict selection probability tends to \(1/2\).

Both selected predictors always have population risk \(1/2\), so the excess risk is exactly zero. Nevertheless, the data do not favor one index with probability tending to one. A fixed tie-breaking rule only resolves exact empirical ties; it cannot remove the sampling-driven switches between strict inequalities.

Common Mistake: Treating Risk Consistency as Model Recovery

Wrong reasoning: “The selected risk approaches the minimum candidate risk, so the selector must eventually choose the best model.” This conclusion silently assumes that a single index is separated from all alternatives by a positive risk gap.

Correction: Risk consistency controls the size of the risk difference, not the identity of the index. Under an exact tie there is no unique risk-best index; under a shrinking gap, the risk difference can vanish even while the selector keeps choosing a non-best index with substantial probability. To infer recovery, specify a unique target and verify a positive separation condition, as in the recovery theorem.

Key takeaway: Risk consistency is about predictive performance relative to a stated comparator. Model-index recovery is a stronger claim that needs an identifiable target, typically a unique minimizer separated by a positive risk gap. Ties and near-ties can preserve excellent risk while making the selected label unstable or unrecoverable.

Check Your Understanding

Use the definitions and results above to answer the following.

In the first worked example, why is the selected predictor risk consistent even though the two prediction functions differ at every input?
State the uniform-deviation event and positive-gap condition that guarantee exact empirical recovery of a unique best index.
In the shrinking-gap example, calculate the population excess risk if the selector chooses index 1.
Why does convergence of \(R_{\hat j}-R_{\min}\) to zero not imply that \(P(\hat j=j_0)\) tends to one?
What does the risk bound for penalized selection require of the penalty attached to a best-risk comparator for it to yield risk consistency?