Three Different Questions About a Fitted Model
When a learning algorithm returns a model with very small training loss, three questions are easy to conflate: How well did the algorithm minimize the training objective? How well does the fitted model predict on new data? And how much of any prediction error is avoidable within the model class? These are different questions, and answering one does not automatically answer the others.
The distinction is especially important in overparameterized models. Such a model may have enough freedom to fit the observed responses exactly, but the training data only constrain its predictions at the observed inputs. What happens at new inputs depends on the data, the model, and the solution selected by the algorithm. A training loss that approaches zero is a statement about the observed sample; by itself, it is not a statement about prediction on future observations.
The empirical-risk infimum is taken over the specified parameter space \(\Theta\). If a minimum is attained, zero optimization error means that the algorithm has reached an empirical-risk minimizer; it does not necessarily mean that the minimum risk is zero. Exact interpolation is the stronger condition \(\widehat R_n(\hat\theta)=0\), when the loss is nonnegative. In contrast, prediction risk is an expectation over new observations. It can be high even when the empirical risk is zero.
It is also useful to compare prediction risk with a reference model \(\theta_0\). The difference \(R(\hat\theta)-R(\theta_0)\) is an excess prediction risk relative to that reference. This comparison is not the optimization error: the reference model need not minimize the empirical objective, and empirical minimization need not reduce prediction risk.
Squared-Error Risk and Unavoidable Response Variation
For squared-error prediction, some prediction error can remain even for the best possible prediction rule. The following identity separates response variation from error due to an imperfect conditional mean prediction.
Proof. Conditional on \(X\), write \(Y-a(X)=(Y-m(X))+(m(X)-a(X))\). Squaring and taking the conditional expectation gives
because \(E[Y-m(X)\mid X]=0\). Taking expectations proves the identity. The finite-second-moment assumptions ensure the terms are integrable. \(\square\)
The first term is the expected conditional variance; no predictor that uses only \(X\) can remove it. The second term measures squared distance from the conditional mean. This identity concerns prediction risk, not the empirical objective, and it does not guarantee that a particular model class can represent \(m(X)\).
Worked Example: Two Possible Responses at One Input
Suppose \(X=e_1\) with probability one, and \(Y\) equals 0 or 2 with equal probability. A linear predictor \(f_w(X)=w^\top X\) therefore predicts the constant \(c=w_1\). Calculate its prediction risk and identify the irreducible contribution.
Direct calculation. The two possible responses have equal probability, so
The minimum is attained at \(c=1\), and the minimum prediction risk is 1. At that prediction, the two weighted squared errors are \(\tfrac12(1-0)^2=\tfrac12\) and \(\tfrac12(1-2)^2=\tfrac12\); their sum is 1, not \(\tfrac12\).
This agrees with the conditional-variance identity: \(E[Y\mid X=e_1]=1\) and \(\operatorname{Var}(Y\mid X=e_1)=1\). Even the optimal prediction has risk 1 because the response varies at the same input. A more flexible model cannot eliminate this population variation merely by optimizing harder.
What Optimization Can Establish
Optimization error measures how far the fitted model is from the best empirical objective value available in its parameter space. It is useful for assessing an algorithm, but it is not a stand-alone guarantee about prediction. A direct algebraic comparison shows what additional information is needed.
Proof. Add and subtract the empirical risks at \(\hat\theta\) and \(\theta_0\):
The first and third terms are each at most \(\delta\), by the uniform-deviation assumption. The middle term is at most \(\varepsilon\), by the empirical-risk condition. Adding these bounds proves the claim. This is a deterministic implication on the stated event; a probabilistic guarantee requires a separate argument establishing the uniform-deviation bound. \(\square\)
This result clarifies the roles of the quantities. A small \(\varepsilon\) controls optimization relative to the reference model on the training sample. A small \(\delta\) controls how well empirical risks represent population risks across the parameter space. Without control of the latter, optimizing the empirical objective—even exactly—does not establish a useful prediction-risk comparison.
Worked Examples: Zero Training Loss and Prediction Risk
Worked Example: Fitting a Random Label Can Increase Risk
Let \(X=1\) always and let \(Y\) equal \(+1\) or \(-1\) with probability \(1/2\) each. Use the scalar predictor \(f_\theta(X)=\theta\) and squared-error loss. Suppose the observed training labels are all \(+1\), an event with probability \(2^{-n}\). Starting at \(\theta_0=0\), follow gradient descent on the empirical risk with step size \(0<\eta<1\).
On this sample, \(\widehat R_n(\theta)=(\theta-1)^2\), so the gradient-descent update is
Starting from zero, the iterates are \(\theta_t=1-(1-2\eta)^t\). Since \(|1-2\eta|<1\), they converge to 1. Thus \(\widehat R_n(\theta_t)\to0\). But for a new observation from the same distribution,
The initial predictor has risk \(R(0)=1\), whereas the interpolating limit has risk \(R(1)=2\). The training loss decreases to zero, but prediction risk need not increase monotonically along this path; the interpolating limit has greater risk than the initial predictor. The sample event is unusual for large \(n\), but it has positive probability for every finite \(n\); on that event, zero training loss is plainly not evidence of improved prediction.
Worked Example: An Unobserved Direction Remains Unconstrained
Let \(X\) be uniform on \(\{e_1,e_2\}\), let \(Y=0\) always, and use \(f_w(X)=w^\top X\) with squared-error loss. Consider a training sample in which every input equals \(e_1\), an event with probability \(2^{-n}\). Start gradient descent at \(w_0=(1,2)\) with step size \(0<\eta<1\).
For this sample, \(\widehat R_n(w)=w_1^2\), so the gradient is \((2w_1,0)\). The updates are \(w_{1,t+1}=(1-2\eta)w_{1,t}\) and \(w_{2,t+1}=w_{2,t}=2\). Hence \(w_{1,t}\to0\) and the empirical risk tends to zero, but the second coordinate is never changed by the observed data.
For a new input from the stated distribution, the prediction risk is
It starts at \((1+4)/2=2.5\) and converges to \((0+4)/2=2\). The zero predictor \(w=(0,0)\) has risk 0, so the interpolating limit is not the best predictor even though the model can represent that best predictor. Optimization has removed error in the observed direction; it has provided no information about the unobserved one.
Interpreting the Three Quantities
| Quantity | Question it answers | What it does not establish by itself |
|---|---|---|
| Optimization error | How close is the algorithm to the best empirical objective value? | That the empirical minimum is small or that prediction risk is small. |
| Empirical risk | How much loss does the fitted model incur on the observed sample? | How much loss it incurs on new observations. |
| Prediction risk | What is the expected loss on a new observation from the specified distribution? | That the model is close to an empirical optimum. |
A common mistake is to reason, “The optimizer converged, the training loss is zero, therefore the model generalizes well.” The first claim concerns the algorithm, the second concerns the training sample, and the conclusion concerns new observations. Neither claim alone supplies the missing link. The link requires assumptions or evidence connecting empirical and population risk, and it may also depend on which interpolating solution the algorithm selects.
When reviewing a claim about an overparameterized fit, keep the quantities separate: report the achieved empirical risk, state optimization error relative to a defined empirical comparator, and assess prediction risk using an appropriate population argument or held-out evaluation. A model may interpolate and generalize well, interpolate and generalize poorly, or fail to interpolate despite being optimized as far as its objective permits. Exact fit is a property of the sample fit, not a verdict on prediction.
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- If an algorithm has zero optimization error, must its empirical risk be zero? Explain.
- In the random-label example, why does the population risk equal \(\theta^2+1\) rather than \((\theta-1)^2\)?
- For the unobserved-direction example, which coordinate changes during gradient descent, and why does the other remain fixed?
- What extra condition, beyond small optimization error, is used in the risk-comparison theorem?
- For squared-error prediction, identify the two terms in the conditional-variance decomposition and explain which one a predictor cannot remove.