Why Training Error Falls in Nested Model Classes

Why Compare Training Error Across Model Classes?

A model class is a collection of candidate prediction rules. A more flexible class offers more possible fits to the observed data, but flexibility alone does not guarantee that the best fit has lower training error. The decisive condition is nesting: every prediction rule in the smaller class must also be available in the larger class. When both classes are optimized on the same observations using the same loss, this inclusion guarantees that minimum training error cannot increase.

This is a statement about optimized fit to the training data, not about how well predictions work on new observations. The distinction matters: a lower training error is a mathematical consequence of nesting, while the comparison does not establish that the larger class is preferable for prediction.

Definition: For training observations $(x_i,y_i)$, $i=1,\ldots,n$, and a loss function $L(y,\hat y)$, the empirical training loss of a prediction rule $f$ is $$ \widehat L_{\mathrm{train}}(f)=\frac{1}{n}\sum_{i=1}^{n}L\bigl(y_i,f(x_i)\bigr). $$ A model class $\mathcal M$ is a set of candidate prediction rules. Its minimum training loss is $$ \widehat L_{\mathrm{train}}(\mathcal M)=\min_{f\in\mathcal M}\widehat L_{\mathrm{train}}(f), $$ provided the minimum is attained. The classes are nested if $\mathcal M_1\subseteq\mathcal M_2$.

For squared-error regression, the loss for observation $i$ is $L(y_i,\hat y_i)=(y_i-\hat y_i)^2$. The sum of squared errors is $n$ times the average squared error, so both give the same ordering when the sample size is fixed. Whichever training-loss convention is used, it must be applied consistently to both classes.

The Nesting Result

Suppose two model classes use the same training observations, the same loss function, and the same training-loss definition. If every rule in $\mathcal M_1$ is also in $\mathcal M_2$, then

\widehat L_{\mathrm{train}}(\mathcal M_2) \leq \widehat L_{\mathrm{train}}(\mathcal M_1).

The result follows directly from what it means to minimize over a set. If $f_1^*$ is a rule attaining the minimum in $\mathcal M_1$, nesting means $f_1^*$ is also an available candidate in $\mathcal M_2$. The minimum over $\mathcal M_2$ cannot be greater than the loss of that particular candidate.

Find the smaller-class optimum.
Let $f_1^*\in\mathcal M_1$ minimize training loss over $\mathcal M_1$.

Use inclusion.
Because $\mathcal M_1\subseteq\mathcal M_2$, the same rule $f_1^*$ is an admissible candidate in $\mathcal M_2$.

Compare with the larger-class minimum.
A minimum over $\mathcal M_2$ is no greater than the loss of any one of its candidates, including $f_1^*$.

\widehat L_{\mathrm{train}}(\mathcal M_2) \leq \widehat L_{\mathrm{train}}(f_1^*) =\widehat L_{\mathrm{train}}(\mathcal M_1).

If an optimum is not attained, the same conclusion holds using infima: the infimum over a set cannot exceed the infimum over one of its subsets. This formulation matters for some model classes where a best-fitting rule may not exist, even though losses can get arbitrarily close to a lower bound.

Theorem: For nested model classes evaluated on the same training sample with the same loss and the same optimization target, the optimized training loss is non-increasing as the class expands. The result guarantees “no increase,” not “strict decrease.”

Worked Examples

Worked Example: A Constant Fit Compared With a Linear Trend

A hypothetical reserving analyst records four annual development indicators. The observed pairs are $(x_i,y_i)=(0,1),(1,3),(2,5),(3,7)$. Compare the minimum sum of squared errors (SSE) for a constant prediction, $\mathcal M_1=\{f(x)=a\}$, and a straight-line prediction, $\mathcal M_2=\{f(x)=a+bx\}$. Identify the nesting condition and verify the error comparison.

Method. Minimize SSE in each class. Every constant rule is a straight line with $b=0$, so $\mathcal M_1\subseteq\mathcal M_2$.

Constant class. The least-squares constant is the sample mean: $a=(1+3+5+7)/4=16/4=4$. Its predictions are $4,4,4,4$, so its residuals are $-3,-1,1,3$. Therefore, $SSE_1=(-3)^2+(-1)^2+1^2+3^2=9+1+1+9=20$.

Linear class. The line $f(x)=1+2x$ predicts $1,3,5,7$, respectively. Its residuals are all zero, so its SSE is $0^2+0^2+0^2+0^2=0$. Since squared errors cannot be negative, zero is the minimum possible SSE.

The average squared errors are $20/4=5$ for the constant class and $0/4=0$ for the linear class. Thus the larger class has lower optimized training error, consistent with the theorem. The common mistake avoided here is to assume the result follows merely because a line has more parameters; the proof relies on the constant rules being included in the line class.

Worked Example: Adding a Group Indicator

For a hypothetical portfolio review, four records have outcomes $2,4,6,8$, with the first two in group A and the last two in group B. Under squared-error loss, compare an intercept-only model $f=a$ with the model $f=a+bz$, where $z=0$ for group A and $z=1$ for group B. Calculate both minimum SSEs and show why the larger class is nested around the smaller one.

Method. Fit each model by least squares. Setting $b=0$ in the group-indicator model gives every intercept-only prediction, so the classes are nested.

Intercept-only fit. The mean outcome is $(2+4+6+8)/4=20/4=5$. The four residuals are $-3,-1,1,3$; hence $SSE_1=9+1+1+9=20$.

Group-indicator fit. The group A mean is $(2+4)/2=3$, and the group B mean is $(6+8)/2=7$. The model achieves these fitted values with $a=3$ and $b=7-3=4$. Its residuals are $-1,1,-1,1$, so $SSE_2=1+1+1+1=4$.

The average squared errors are $20/4=5$ and $4/4=1$, respectively. The calculations verify non-increase. Do not compare the intercept-only SSE with the larger model evaluated at an arbitrary coefficient choice and call that the larger class’s minimum: the claim concerns the optimized loss.

Worked Example: More Flexibility Without Strict Improvement

A hypothetical team observes $y=(0,1,0)$ at predictor values $x=(-1,0,1)$. Compare the intercept-only class $f(x)=a$ with the linear class $f(x)=a+bx$, using SSE. Determine whether adding the slope strictly reduces the minimum training error.

Method. Find the least-squares fit in both classes and check whether the additional slope can improve the fit. The intercept-only class is nested in the linear class by taking $b=0$.

Intercept-only fit. The mean is $(0+1+0)/3=1/3$. The residuals are $-1/3,2/3,-1/3$, giving $SSE_1=1/9+4/9+1/9=6/9=2/3$.

Linear fit. The sample mean of $x$ is $0$, and the sample mean of $y$ is $1/3$. The cross-product sum for the slope is $(-1)(0-1/3)+0(1-1/3)+(1)(0-1/3)=1/3+0-1/3=0$. The sum of squared deviations in $x$ is $(-1)^2+0^2+1^2=2$, so the least-squares slope is $b=0/2=0$. The fitted line is the same constant $1/3$, and $SSE_2=2/3$.

The optimized errors tie: $SSE_2=SSE_1=2/3$. The slope adds a possible direction of fit, but the observed data do not call for a nonzero slope. The trap is to interpret “cannot increase” as “must decrease”; inclusion permits equality.

Conditions That Make the Comparison Valid

Before using the result, check that the comparison is genuinely between nested classes and that the training objective has not changed. The word “larger” should mean set inclusion, not simply a higher parameter count or a more complicated label. A model class with more parameters may be nonnested if its restrictions differ from those of the smaller class.

Comparison feature	Required for the result?	Why it matters
Same training observations	Yes	The two empirical losses must be calculated on the same sample.
Same loss and weighting	Yes	The smaller-class optimum must be a candidate under the larger-class objective.
Nested candidate rules	Yes	The smaller-class optimum must remain available in the larger class.
More parameters	No	Parameter count alone does not establish inclusion.
Strictly smaller error	No	The larger class may contain the old optimum without improving on it.

For example, if a model is optimized by squared error and then compared with another model’s absolute error, the numerical values are not the same objective. Likewise, adding a penalty to one objective changes the quantity being minimized: an unpenalized training-error ordering does not by itself order penalized objective values. If software uses an approximate optimizer, it may also fail to find the true minimum; a reported increase can reflect optimization failure rather than a contradiction of the theorem.

Common Exam Trap: Complexity Is Not Nesting

A common incorrect argument is: “Class B is called more complex, so its optimized training error must be no higher.” That conclusion does not follow from the name or parameter count. Consider two classes for observations $y=(0,0)$, both evaluated by SSE. Let $\mathcal A$ contain only the constant prediction $f(x)=0$, and let $\mathcal B$ contain only the constant prediction $f(x)=1$. The losses are

SSE(\mathcal A)=0^2+0^2=0,\qquad SSE(\mathcal B)=(0-1)^2+(0-1)^2=2.

The calculation does not violate the nesting result: $\mathcal A$ is not a subset of $\mathcal B$. Calling a class “more complex” cannot replace checking inclusion. The correction is to write down the candidate sets or model equations and verify that every prediction rule in the smaller class can be reproduced in the larger one.

Key takeaway: With a fixed training sample and a fixed loss, expanding a model class by inclusion cannot raise the minimum training loss. It may lower that loss or leave it unchanged. No ordering is guaranteed when the classes are nonnested or the objective changes.

Check Your Understanding

For each problem, compare optimized training losses only when the stated conditions justify doing so.

A constant model is fitted to outcomes $2,4$, and a linear model is fitted to the same outcomes at predictor values $0,1$, using SSE. Calculate the constant model’s SSE and the linear model’s SSE. (Answer: constant SSE $=2$; linear SSE $=0$.)
Three observations have $z=(0,0,1)$ and $y=(1,1,5)$. Compare the intercept-only model with the intercept-plus-$z$ model under SSE. (Answer: intercept-only SSE $=32/3$; intercept-plus-$z$ SSE $=0$.)
For $x=(-1,0,1)$, the observed outcomes are $y=(2,0,2)$. Under SSE, an intercept-only class is expanded to the class of straight lines. Find both minimum SSEs and state whether the decrease is strict. (Answer: both minimum SSEs are $8/3$; there is no strict decrease.)
A model class $\mathcal M_1$ is contained in $\mathcal M_2$. Both are fit to the same observations using the same weighted squared-error loss, with identical fixed positive weights. What ordering must hold for their minimum weighted training losses? (Answer: the minimum loss for $\mathcal M_2$ is less than or equal to that for $\mathcal M_1$.)
Two classes are described as having two and five parameters, respectively, but no model equations or inclusion relationship are given. Can the minimum training errors be ordered from this information alone? (Answer: no; parameter counts alone do not establish nesting.)

What You'll Learn