Why Better Fit Is Not the Only Goal
A regression model summarizes how a response varies with predictors. Adding another predictor gives the model an additional way to match the observed responses. That flexibility can improve the fit to the data already observed, but it also makes the model more complex. A model that fits the observed data especially closely is not automatically the most useful model: some of its added flexibility may capture accidental features of this particular sample.
This tutorial develops that tension using two basic quantities: residual sum of squares, which measures discrepancies between observed and fitted responses, and the number of fitted coefficients, a simple way to describe model complexity. The key result is that the residual sum of squares cannot increase when a predictor is added to a nested least-squares model fitted to the same observations. That result explains why fit alone tends to favor more elaborate models; it does not tell us that the most elaborate model is best.
The Comparison Must Be Fair
To explain the fit-complexity tradeoff, consider ordinary least squares (OLS): for a specified model, the fitted coefficients are chosen to minimize the sum of squared residuals. A model is nested within another model if every fitted-value pattern the smaller model can produce is also available to the larger model. For example, a model with an intercept and predictor \(x\) is nested within one with an intercept and predictors \(x\) and \(z\): the larger model can reproduce the smaller model by setting the coefficient of \(z\) to zero.
A meaningful comparison of their RSS values uses the same response, the same observations, and the same least-squares fitting method. If models use different observations, their RSS values sum squared errors over different sets of responses and are not a direct comparison of fit. If two models are not nested, there is no general guarantee that one must have a smaller RSS merely because it has more terms. Even with nested models, a term that adds no independent direction to the model cannot increase the number of estimable coefficients or improve the fitted values.
The monotonicity result below is about least-squares fitting, not about a probability model for the errors. No normality assumption is needed for the result. Additional assumptions may matter for other questions, such as inference or prediction, but they are not required to show that the minimum RSS over a larger set of possible fits cannot exceed the minimum over a smaller set.
Why the RSS Cannot Increase
A short least-squares argument establishes the result. Think of the \(n\) observed responses as a vector \(\mathbf{y}\). Each model permits fitted-value vectors in a particular set of possible directions. Nesting means that the smaller model's set of fitted values is contained in the larger model's set. Least squares chooses, from its permitted set, the fitted-value vector closest to \(\mathbf{y}\), where closeness is measured by squared Euclidean distance. The larger model can always use the smaller model's best fit, so its minimum distance cannot be greater.
The same point can be made using residuals. Write \(\hat{\mathbf{y}}_S\) for the smaller model's fitted vector and \(\hat{\mathbf{y}}_L\) for the larger model's. The difference \(\hat{\mathbf{y}}_L-\hat{\mathbf{y}}_S\) is a change to the fit allowed by the larger model. Least squares in the larger model makes its residual vector \(\mathbf{y}-\hat{\mathbf{y}}_L\) orthogonal to every allowed change in fitted values. Thus the smaller model's residual is the sum of the larger model's residual and that fitted-value change, with the two parts orthogonal.
The squared length of the fitted-value change is nonnegative, which gives the inequality. It is zero when the two fitted vectors are identical; otherwise, the larger model has a strictly smaller RSS. This justifies the fit part of the tradeoff. The added flexibility can reduce the observed residual error, while increasing the number of estimated coefficients.
Worked Example: Adding a Predictor Reduces RSS
Suppose a hypothetical data set contains four observations. Compare an intercept-only model with a model that adds \(x\).
| Observation | \(x\) | \(y\) |
|---|---|---|
| 1 | 0 | 1 |
| 2 | 1 | 2 |
| 3 | 2 | 2 |
| 4 | 3 | 5 |
For the intercept-only model, the fitted value for every observation is the response mean, \(\bar{y}=10/4=2.5\). Its RSS is $$ \mathrm{RSS}_{\text{intercept}} =(1-2.5)^2+(2-2.5)^2+(2-2.5)^2+(5-2.5)^2 =2.25+0.25+0.25+6.25=9. $$
For the model with an intercept and \(x\), the sample means are \(\bar{x}=1.5\) and \(\bar{y}=2.5\). The least-squares slope is \(6/5=1.2\), since \(\sum (x_i-\bar{x})(y_i-\bar{y})=6\) and \(\sum (x_i-\bar{x})^2=5\). The intercept is \(2.5-(1.2)(1.5)=0.7\), so the fitted equation is \(\hat{y}=0.7+1.2x\). Its fitted values are \(0.7,1.9,3.1,4.3\), giving $$ \mathrm{RSS}_{x} =(1-0.7)^2+(2-1.9)^2+(2-3.1)^2+(5-4.3)^2 =0.09+0.01+1.21+0.49=1.8. $$
Both models use the same four responses, and the intercept-only model is nested within the model with \(x\). Adding \(x\) lowers the RSS from \(9\) to \(1.8\), while increasing the coefficient count from one to two. This is an in-sample fit improvement; the calculation alone does not establish whether the added complexity is worthwhile.
How Much Does an Added Predictor Explain?
When a new predictor is added to a nested OLS model, the decrease in RSS corresponds to the amount of residual variation that the new model can fit. If the predictor is unrelated to the smaller model's residual direction in the least-squares geometry, the fitted values may not change. If it aligns with that residual variation, the RSS decreases. The size of the decrease varies: an added term can account for a large part of the remaining residual sum of squares or only a small part.
Worked Example: A New Term Captures Residual Variation
Consider five hypothetical observations described by two predictors, \(x\) and \(z\), and response \(y\).
| Observation | \(x\) | \(z\) | \(y\) |
|---|---|---|---|
| 1 | -2 | 1 | 7 |
| 2 | -1 | -2 | 3 |
| 3 | 0 | 0 | 8 |
| 4 | 1 | 2 | 17 |
| 5 | 2 | -1 | 15 |
Compare the model with an intercept and \(x\) against the model that also includes \(z\). The predictor columns have mean zero, \(\sum x_i^2=10\), \(\sum z_i^2=10\), and \(\sum x_i z_i=0\). The response can be written row by row as \(y_i=10+3x_i+2z_i+e_i\), where the remaining values are \(e=(1,0,-2,0,1)\). These values have \(\sum e_i=0\), \(\sum x_i e_i=0\), and \(\sum z_i e_i=0\).
Consequently, the model with both predictors has fitted equation \(\hat{y}=10+3x+2z\), with residuals \(1,0,-2,0,1\). Its RSS is \(1^2+0^2+(-2)^2+0^2+1^2=6\). If \(z\) is omitted, the fitted equation is \(\hat{y}=10+3x\). Its residuals are \(2z_i+e_i\), namely \(3,-4,-2,4,-1\), so $$ \mathrm{RSS}_{x} =3^2+(-4)^2+(-2)^2+4^2+(-1)^2 =9+16+4+16+1=46. $$
Including \(z\) reduces the RSS by \(46-6=40\). The model with \(x\) has two coefficients, including its intercept; the model with \(x\) and \(z\) has three. This example shows a substantial fit improvement from the added term, but it still does not establish whether that improvement is worth the extra complexity for the intended use.
A Smaller Improvement Can Still Cost a Coefficient
The key tradeoff is especially clear when the RSS decreases but only modestly. An added coefficient is still part of the fitted model even if the reduction in observed residual error is small. Whether to retain the term cannot be settled by checking only that the RSS went down; the decision also depends on the purpose of the model and an appropriate way to assess complexity.
Worked Example: A Further Term Gives a Modest Reduction
Return to the four observations in the first example. The fitted equation with \(x\) had residuals \(0.3,0.1,-1.1,0.7\) and RSS \(1.8\). Now consider adding a second predictor with values \(z=(1,-1,-1,1)\). This predictor has \(\sum z_i=0\), \(\sum z_i x_i=0\), and \(\sum z_i^2=4\), so it adds a direction independent of the intercept and \(x\).
The residuals from the model with \(x\) have dot product with \(z\) equal to $$ (0.3)(1)+(0.1)(-1)+(-1.1)(-1)+(0.7)(1)=2. $$ Thus the least-squares coefficient for \(z\), after accounting for the intercept and \(x\), is \(2/4=0.5\). The new residuals are the old residuals minus \(0.5z\): \(-0.2,0.6,-0.6,0.2\). The new RSS is $$ (-0.2)^2+(0.6)^2+(-0.6)^2+(0.2)^2 =0.04+0.36+0.36+0.04=0.8. $$
The second predictor reduces RSS by \(1.8-0.8=1.0\), while raising the coefficient count from two to three. The RSS decreases, as the nesting result requires, but the reduction is much smaller than the first predictor's reduction from \(9\) to \(1.8\). The arithmetic describes the in-sample tradeoff; deciding whether this further term is useful requires more than the fact that \(0.8<1.8\).
What the Tradeoff Does—and Does Not—Tell Us
RSS and parameter count answer related but different questions. RSS reports how closely a model fits the observed responses, in squared response units. Parameter count reports how many independently fitted regression coefficients are used to obtain that fit. Neither quantity alone is a complete measure of model quality. A very small RSS can result from fitting sample-specific variation, and a low coefficient count does not guarantee that the model captures the relationships needed for its purpose.
Model specification should still be guided by the scientific or forecasting question, predictor availability, and a defensible set of candidate models. Once candidates have been specified, fit and complexity can be considered together. Formal comparison measures that penalize complexity are introduced separately; this tutorial establishes why such a balance is needed without choosing a particular measure.
Common Mistakes
- “The model with the lowest RSS must be best.” Among nested models fitted by OLS to the same observations, RSS cannot increase as terms are added. That makes RSS alone tend to favor the largest candidate, not necessarily the most useful one. Consider complexity and the model's intended purpose.
- “The RSS decrease proves the new predictor matters scientifically.” The decrease is a statement about fit to the observed data. It does not, by itself, establish a causal effect, generalizable predictive value, or a sound scientific reason to include the term.
- “Any two RSS values can be compared directly.” The monotonicity result requires nested models fitted to the same response observations by least squares. Comparisons across different samples or between non-nested models do not inherit that guarantee.
- “Every added column counts as one more parameter.” A predictor that is exactly redundant with existing terms does not add an independently estimable coefficient. Count the model's estimable coefficients, not just the number of named terms.
Check Your Understanding
Answer each question using the definitions and results developed above.
- What does RSS measure, and in what units is it expressed if the response is measured in centimeters?
- Why can the minimum RSS of a larger nested OLS model not exceed that of a smaller model fitted to the same observations?
- A candidate model adds a predictor that is an exact linear combination of existing predictors. Does it necessarily add an independently estimable coefficient? Explain.
- Why is a lower RSS not, by itself, enough to conclude that an added predictor should be retained?
- Name two conditions needed for the nested-model RSS guarantee to apply.