Risk Consistency Versus Parameter Consistency

Two Different Notions of Consistency

A learning procedure produces a predictor, often by estimating a parameter and substituting it into a model. Two questions then arise: does the predictor’s population risk approach the best available risk, and does its parameter approach a particular target? These questions concern different objects. Risk evaluates predictions under the data distribution; parameter convergence concerns coordinates used to represent a predictor. A small risk gap alone need not determine those coordinates.

Let \(Z=(X,Y)\) have the population distribution, let \(h_\theta\) be a predictor indexed by \(\theta\in\Theta\), and let \(\ell\) be a loss. For parameters with finite expected loss, define the population risk by \(R(\theta)=E[\ell(Y,h_\theta(X))]\), and write \(R_*=\inf_{\theta\in\Theta}R(\theta)\). An estimator \(\hat\theta_n\) is random because it is computed from a sample of size \(n\); its fitted predictor is \(h_{\hat\theta_n}\).

Definition: The estimator is risk consistent if \(R(\hat\theta_n)-R_*\xrightarrow{p}0\). Given a specified target parameter \(\theta_*\), it is parameter consistent for that target if \(\hat\theta_n\xrightarrow{p}\theta_*\). Risk consistency is a statement about the loss of the fitted predictor; parameter consistency is a statement about the location of the estimated parameter.

The distinction matters even when a risk minimizer exists. Different parameter values can represent the same predictor, or risks can change very little as a parameter moves. In either case, good population performance may coexist with parameters far from a chosen target. The examples below make these possibilities explicit.

Worked Examples

Worked Example: Diverging Parameters, Identical Predictions

Suppose \(X\sim N(0,1)\), \(Y=X\), and the model is \(h_{(a,b)}(x)=(a+b)x\), with squared-error loss. Take the target parameter to be \(\theta_*=(1,0)\). For any \((a,b)\), the population risk is

R(a,b)=E\left[(Y-h_{(a,b)}(X))^2\right] =E\left[(X-(a+b)X)^2\right] =(a+b-1)^2E[X^2] =(a+b-1)^2.

The last equality uses \(E[X^2]=1\). In particular, \(R_*=0\), attained whenever \(a+b=1\). Now let the estimator return \(\hat\theta_n=(n,1-n)\). For every \(n\), its two coordinates sum to \(1\), so

R(\hat\theta_n)=(n+1-n-1)^2=0, \qquad \|\hat\theta_n-\theta_*\|_2 =\|(n-1,1-n)\|_2 =\sqrt{2}(n-1).

Thus the excess risk is always zero, while the distance from the target tends to infinity. The predictions are identical to those from \(\theta_*\); the parameter coordinates are not. Risk consistency therefore does not, by itself, select a parameter representation.

Worked Example: Risk Approaches Its Minimum Along a Diverging Sequence

Let \(Y=0\) with probability one, and use the constant predictor \(h_\theta(x)=\theta/(1+\theta^2)\), where \(\theta\in\mathbb R\). Under squared-error loss the risk is

R(\theta)=\left(\frac{\theta}{1+\theta^2}\right)^2 =\frac{\theta^2}{(1+\theta^2)^2}.

Because a squared quantity is nonnegative and \(R(0)=0\), the minimum risk is \(R_*=0\). Consider the deterministic sequence \(\hat\theta_n=n\). Its excess risk is

R(\hat\theta_n)-R_*= \frac{n^2}{(1+n^2)^2} \leq \frac{1}{n^2} \longrightarrow 0, \qquad |\hat\theta_n|=n\longrightarrow\infty.

The inequality follows from \((1+n^2)^2\geq n^4\). The predictions approach zero, which is optimal, even though the parameter escapes to infinity. Unlike the first example, these parameters do not all give exactly the same prediction: their predictions get closer to the optimum as \(n\) increases.

Worked Example: Exact Risk Consistency Without Convergence to a Point

Let \(Y=1\) with probability one and let the constant predictor be \(h_\theta(x)=\theta^2\), again under squared-error loss. Then

R(\theta)=(1-\theta^2)^2.

The minimum \(R_*=0\) is attained at both \(\theta=1\) and \(\theta=-1\). Set \(\hat\theta_n=(-1)^n\). For every \(n\), \(\hat\theta_n^2=1\), so \(R(\hat\theta_n)-R_*=0\). But the sequence alternates between \(1\) and \(-1\), and hence does not converge to either point: its even and odd subsequences have different limits. For the specified target \(\theta_*=1\), the distance is \(0\) for even \(n\) and \(2\) for odd \(n\), so it does not tend to zero.

This is not a failure of prediction: every fitted predictor is exactly optimal. It is a failure to converge to one particular parameter representative. The target must be specified before parameter consistency has a definite meaning.

Why Equal Predictors Have Equal Risk

The first and third examples use different parameter values to represent the same predictions. The following elementary result explains why such representations cannot be distinguished by population risk for a loss that depends on the data only through the prediction.

Proposition: Suppose \(h_\theta(X)=h_\eta(X)\) with probability one under the marginal distribution of \(X\), and suppose the losses \(\ell(Y,h_\theta(X))\) and \(\ell(Y,h_\eta(X))\) are integrable. Then \(R(\theta)=R(\eta)\).

Proof. The event \(A=\{h_\theta(X)=h_\eta(X)\}\) has probability one. On \(A\), applying the same loss function to the same \(Y\) and the same prediction gives

\ell(Y,h_\theta(X))=\ell(Y,h_\eta(X)).

The two loss random variables are therefore equal with probability one. Integrable random variables that are equal with probability one have equal expectations, so \(E[\ell(Y,h_\theta(X))]=E[\ell(Y,h_\eta(X))]\), which is \(R(\theta)=R(\eta)\). \(\square\)

This result concerns equality of predictors on the population support of \(X\). Predictors may differ at inputs that have probability zero without changing their population risk. It also shows why risk alone cannot tell apart two parameters that produce the same predictions where the data occur.

What Excess Risk Says About Predictions

Risk consistency is not empty information: under suitable loss and target conditions, it can imply convergence of predictions in a population-average sense. For squared-error loss, the conditional mean gives an exact decomposition. This clarifies what risk consistency controls, without turning that statement into a claim about parameter coordinates.

Proposition: Suppose \(E[Y^2]<\infty\), and define \(f_0(X)=E[Y\mid X]\). For any measurable predictor \(f(X)\) with \(E[f(X)^2]<\infty\),

$$ E[(Y-f(X))^2] = E[(Y-f_0(X))^2]+E[(f(X)-f_0(X))^2]. $$

Proof. Write \(Y-f=(Y-f_0)+(f_0-f)\), square, and take expectations:

$$ E[(Y-f)^2] =E[(Y-f_0)^2]+E[(f-f_0)^2] +2E[(Y-f_0)(f_0-f)]. $$

The cross term is zero. In fact, \(f_0-f\) is a function of \(X\), so the defining property of conditional expectation gives

E[(Y-f_0)(f_0-f)] =E\left[E[Y-f_0\mid X](f_0-f)\right] =E[0\cdot(f_0-f)] =0.

All expectations are finite: \(Y\), \(f_0\), and \(f\) have finite second moments, and products of square-integrable variables are integrable by the Cauchy–Schwarz inequality. Substitution proves the displayed identity. \(\square\)

Consequently, when the benchmark is the conditional-mean predictor \(f_0\), excess squared-error risk equals \(E[(f(X)-f_0(X))^2]\). For a random fitted predictor, the same identity holds conditional on the training data, provided the evaluation pair \((X,Y)\) is independent of that data and the required second moments are finite. Thus excess-risk convergence in probability gives convergence of this population-average prediction error in probability. It still does not say that the parameter used to produce \(f\) converges to any particular value.

Common Mistake and Practical Interpretation

Wrong reasoning: “The excess risk tends to zero, so \(\hat\theta_n\) must tend to the true parameter.” This conclusion assumes a link between risk difference and distance from that parameter. Risk consistency alone supplies no such link: the first example has a whole line of equally good parameter values, and the second has nearly optimal risk at parameters of increasing magnitude.

Correction: First state which object is being assessed. Excess risk compares population performance with a benchmark. Parameter consistency compares an estimator with a specified parameter target. To pass from the former to the latter requires additional structure linking parameter distance to risk difference; without it, the examples show that the implication can fail.

This distinction is useful when interpreting learning guarantees. A guarantee about excess risk can be valuable even when parameter estimates are unstable or nonunique, especially if the scientific aim is prediction. If the aim is to interpret or recover parameters, a small risk bound is not enough on its own. One must also ask whether different parameter values produce distinguishable predictions under the population distribution and whether risk worsens sufficiently when the parameter moves away from its target.

Key takeaway: Risk consistency controls population performance, not parameter coordinates. It may imply prediction convergence under additional conditions on the loss and target, but parameter consistency requires a separate justification.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

In the linear-model example, why is the excess risk zero for every \(n\) even though the parameter distance grows?
For \(R(\theta)=\theta^2/(1+\theta^2)^2\), verify that \(R(n)\leq 1/n^2\) for positive integers \(n\). What does this bound establish?
In the model \(h_\theta=\theta^2\), why does the alternating estimator fail to converge to \(1\) even though its risk is always minimal?
State the condition on two predictors in the proposition that guarantees equal population risk.
Under the squared-loss decomposition, what quantity equals excess risk over the conditional-mean predictor? Does it directly measure parameter distance?