Residual Sum-of-Squares Reduction in Nested Models

What Does a Larger Regression Model Gain?

When a regression model is expanded by adding predictors, its fitted values can adjust to the data in more ways. A direct way to describe the resulting change in fit is to compare the models’ residual sums of squares. This comparison asks how much less squared residual error the expanded model leaves on the observations used to fit both models.

The focus here is the size and interpretation of that reduction. It is an absolute amount measured in squared outcome units, not a percentage and not, by itself, evidence that the expanded model is useful. To make the comparison meaningful, the two models must be fitted to the same outcome values for the same observations.

Definition: For observations $y_1,\ldots,y_n$ and fitted values $\hat{y}_1,\ldots,\hat{y}_n$, the residual sum of squares is $$ RSS=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2. $$ For a reduced model $R$ and a full model $F$, define the reduction in residual sum of squares as $$ \Delta RSS=RSS_R-RSS_F. $$ A positive $\Delta RSS$ means that the full model has a smaller residual sum of squares on these observations; a value of zero means the two fitted models have the same RSS.

Nested Models and the Fit Comparison

A full model is nested with respect to a reduced model when the reduced model can be obtained from the full model by setting one or more of the full model’s coefficients to specified values, usually zero. For example, an intercept-only model is nested in a model with an intercept and one predictor: setting the predictor’s coefficient to zero gives the reduced model.

The nesting condition matters because the full model’s set of possible fitted values contains all the fitted values available to the reduced model. When each model is fitted by least squares, the reduced model’s coefficients are chosen to minimize its RSS, and the full model’s coefficients are chosen to minimize the full model’s RSS. Since the full model can reproduce any reduced-model fit, its minimum cannot be larger.

RSS_F\leq RSS_R \qquad\text{and therefore}\qquad \Delta RSS=RSS_R-RSS_F\geq 0.

This is a mathematical property of least-squares fitting for nested models, not a promise that the reduction will be large. The full model may fit substantially better on the observed data, improve RSS only slightly, or produce exactly the same fitted values and RSS. If a calculation gives $RSS_F>RSS_R$, check that both models were fitted by least squares, are genuinely nested, and use the same observations and outcome.

Calculating the Reduction

The calculation is a subtraction, but first be clear about what each RSS represents. For each model, subtract its fitted value from the observed outcome for every observation, square each residual, and add the squared residuals. Then subtract the full-model RSS from the reduced-model RSS. A reported RSS can be used directly, provided both values come from the same data and use the same outcome scale.

Worked Example: Subtracting Reported RSS Values

Suppose a hypothetical analysis uses the same 48 observations and the same outcome, a laboratory measurement in milligrams per litre. The reduced model has $RSS_R=126.4\;(\text{mg/L})^2$, and the nested full model has $RSS_F=91.7\;(\text{mg/L})^2$. Calculate and interpret the reduction.

By definition, $$ \Delta RSS=RSS_R-RSS_F. $$ Substituting the reported values gives $$ \Delta RSS=126.4-91.7=34.7\;(\text{mg/L})^2. $$

On these same 48 observations, the full model leaves 34.7 squared milligrams-per-litre less residual error than the reduced model. The unit is squared because each residual, measured in mg/L, is squared before the residuals are added. This number does not say that the full model’s typical prediction error is 34.7 mg/L; it is a difference between sums of squared errors.

The RSS values may also be calculated from residuals directly. This is especially useful for seeing what is being subtracted: each model has its own set of residuals, and its RSS is the sum of their squares. The difference compares the two totals; it is not generally found by subtracting corresponding residuals and then squaring those differences.

Worked Example: Comparing an Intercept-Only Model With a Line

Consider hypothetical observations of study time $x$, in hours, and quiz score $y$, in points:

Observation	$x$ (hours)	$y$ (points)
1	0	2
2	1	4
3	2	5
4	3	7

Compare the reduced intercept-only model with the full linear model that includes $x$. The mean score is $\bar{y}=4.5$, so the fitted value from the reduced model is 4.5 for each observation. Its residuals are $-2.5,-0.5,0.5,2.5$, giving $$ RSS_R=(-2.5)^2+(-0.5)^2+(0.5)^2+(2.5)^2 =6.25+0.25+0.25+6.25=13.00\;\text{points}^2. $$

For these data, the least-squares line is $\hat{y}=2.1+1.6x$. Its fitted values at $x=0,1,2,3$ are $2.1,3.7,5.3,6.9$, so the residuals are $-0.1,0.3,-0.3,0.1$. Thus $$ RSS_F=(-0.1)^2+(0.3)^2+(-0.3)^2+(0.1)^2 =0.01+0.09+0.09+0.01=0.20\;\text{points}^2. $$

The intercept-only model is nested in the line model: setting the slope to zero gives a constant fitted value. The reduction is $$ \Delta RSS=13.00-0.20=12.80\;\text{points}^2. $$

Therefore, on these four observations, fitting the line reduces the sum of squared residuals by 12.80 points squared compared with fitting only a constant. This is a comparison of in-sample sums of squares; it is not a claim that study time caused the observed scores.

Why the Reduction Cannot Be Negative

The nonnegative reduction follows directly from the least-squares minimization, without requiring a probability model for the observations. Let $\mathcal{B}_R$ be the set of coefficient choices allowed by the reduced model and $\mathcal{B}_F$ the set allowed by the full model. Nesting means that every reduced-model fit can also be made by some choice in the full model, so $\mathcal{B}_R$ is contained in $\mathcal{B}_F$.

The reduced-model RSS is the smallest sum of squared residuals over $\mathcal{B}_R$; the full-model RSS is the smallest over $\mathcal{B}_F$. Minimizing over a set that includes all the choices in the smaller set cannot produce a worse minimum. Formally,

RSS_F =\min_{\beta\in\mathcal{B}_F}\sum_{i=1}^{n}(y_i-\hat{y}_i(\beta))^2 \leq \min_{\beta\in\mathcal{B}_R}\sum_{i=1}^{n}(y_i-\hat{y}_i(\beta))^2 =RSS_R.

This argument justifies the ordering for least-squares fits of nested models. It does not establish that the added predictors describe an important relationship, that the result would hold for new observations, or that the change is statistically significant. Those are different questions requiring additional reasoning.

Worked Example: Measuring a Block Addition

Suppose a hypothetical analysis of the same 72 patients and the same outcome—recovery time in days—compares a reduced model with a full model that adds a block of two predictors. The reduced model’s residuals have squared values that sum to $RSS_R=308.5\;\text{days}^2$. The full model’s squared residuals sum to $RSS_F=274.2\;\text{days}^2$. Calculate the gain.

The full model contains the reduced model as the special case in which both added-predictor coefficients are zero. The comparison is therefore nested. The arithmetic is $$ \Delta RSS=308.5-274.2=34.3\;\text{days}^2. $$

The expanded model reduces the total squared residual error by 34.3 days squared on these 72 patients. The reduction is for the added block as a whole. This calculation alone does not assign separate amounts of the gain to the two predictors, nor does it tell us whether the gain is large relative to a relevant benchmark. It reports an observed in-sample difference in RSS.

What the Fit Gain Does and Does Not Measure

The reduction is expressed in squared outcome units. If the outcome is measured in dollars, the RSS reduction is in dollars squared; if the outcome is measured in kilograms, the reduction is in kilograms squared. It is an absolute difference between totals, so its numerical size depends on the outcome’s scale and on how many observations are included.

The comparison is about the data used to fit the models. For a direct comparison, both models must use the same response values and the same observations. If one model is fitted to a different subset because of missing data, the RSS totals combine changes in model and changes in data. Their subtraction would not isolate the fit difference for a common set of observations.

A large reduction does not automatically mean the added predictors are practically important. Nor does a small reduction prove that they are irrelevant: the interpretation depends on the outcome, its units, the scale of the RSS, and the purpose of the analysis. The reduction also does not by itself measure how well the model predicts cases that were not used in fitting.

For a numerical check, one may verify that $RSS_R-RSS_F$ matches the difference between the two reported totals and that the result is nonnegative under the stated conditions. Always retain the outcome’s squared units. Do not relabel the result as a percentage or as an average error unless a separate calculation defines that quantity.

Common Mistakes

Reversing the subtraction. Writing $RSS_F-RSS_R$ gives a nonpositive number under ordinary least-squares nesting. The reduction is defined as reduced-model RSS minus full-model RSS.
Comparing different data sets. RSS values from different observations do not isolate the effect of adding predictors. Confirm that the same outcome and cases were used.
Dropping the square from the units. Residuals are in outcome units, but RSS and its reduction are in squared outcome units.
Treating the reduction as a causal effect or a prediction guarantee. It describes the change in fit on the data used, not a cause-and-effect relationship or performance on new data.
Assuming any expanded model must have lower RSS. The nonnegative result requires least-squares fitting and genuine nesting. Models that are not nested, use different fitting criteria, or use different observations need not follow that ordering.

Key takeaway: For least-squares fits of nested models on the same outcome and observations, $\Delta RSS=RSS_R-RSS_F$ is nonnegative. It measures the decrease in the in-sample sum of squared residuals, in squared outcome units, and does not by itself establish practical value, statistical significance, causation, or predictive performance.

Check Your Understanding

Answer each question using the definition and conditions from this tutorial.

A reduced model has $RSS_R=455$ and a nested full model has $RSS_F=421$, both measured using the same response and cases. What is $\Delta RSS$, and what are its units if the response is measured in millimetres?
In one or two sentences, explain why least-squares RSS cannot increase when a genuinely nested model is expanded.
A full model has a smaller RSS than a reduced model, but it was fitted after excluding 12 observations. Why does this subtraction not isolate the effect of adding predictors?
Does a positive reduction in RSS prove that the added predictors improve predictions for new observations? Explain.
If a response is measured in kilograms, what units should be reported for the reduction in RSS, and why?