Introduction
When a least-squares regression model gains a predictor, it gains another way to fit the observed outcomes. It can use that predictor, or assign it a coefficient of zero and retain the old fit. As a result, adding predictors to a nested model cannot make its in-sample ordinary R-squared smaller. This mathematical guarantee is useful, but it also creates a limitation: the largest R-squared among candidate models will favor the most flexible fit, without charging it for its added complexity.
This tutorial establishes that result and shows what it does, and does not, tell us. The comparison concerns ordinary R-squared calculated on the same outcome values for models fitted by least squares. It does not require a probability model for the errors.
Definition and Conditions for Comparison
The definition uses one fixed set of outcomes, so \(SST\) is the same for every candidate model in a valid comparison. With an intercept, the least-squares fit includes the intercept-only model as a possible fit, and \(0\leq R^2\leq 1\). If all outcomes are identical, \(SST=0\), so this definition of \(R^2\) is undefined. The results below concern the usual intercept-containing models; formulas and behavior can differ for regressions without an intercept.
A model is nested in another model when the larger model can reproduce every fitted-value pattern available to the smaller one. In ordinary linear regression, this holds when the columns of the smaller model's design matrix are included among the columns of the larger model's design matrix. For instance, an intercept-and-age model is nested in an intercept-age-and-exercise model. The observations, outcome, and response scale must be the same for their ordinary R-squared values to be directly compared.
Why R-Squared Cannot Decrease in a Nested Model
Least squares chooses coefficients that minimize the sum of squared residuals among the fits permitted by the model. Suppose the smaller model has \(SSE_{\text{small}}\), and a larger nested model adds one or more predictors. In the larger model, set all newly added coefficients to zero and keep the smaller model's coefficients unchanged. This reproduces the smaller model's fitted values and its sum of squared residuals. Since least squares in the larger model chooses the best fit available there, its minimized residual sum of squares cannot exceed that reproduced value.
For the same observations and outcomes, \(SST\) does not change. If the outcomes are not all identical, \(SST>0\), so dividing the inequality by \(SST\) preserves its direction. Subtracting from 1 reverses the direction of the inequality:
This is a property of optimization over nested sets of possible fits, not a claim that each added predictor is scientifically useful or that the larger model will perform better on new observations. No normality, constant variance, or independence assumption is needed for this algebraic result. The comparison does require least-squares fits, nesting, the same observed outcomes, and a defined common \(SST\).
Worked Example: An Added Predictor Raises R-Squared
Consider four hypothetical locations with an outcome \(y\), a predictor \(x\), and a second candidate predictor \(z\). The values are constructed to make the arithmetic transparent; they do not represent a real data set.
The outcomes are \(y=(1,2,2,5)\), and the first predictor values are \(x=(0,1,2,3)\). The outcome mean is \(\bar{y}=2.5\), so
For the intercept-and-\(x\) model, the least-squares slope is \(1.2\) and the intercept is \(0.7\), giving fitted values \((0.7,1.9,3.1,4.3)\). The residuals are \((0.3,0.1,-1.1,0.7)\), hence
Now add the predictor \(z=(3,1,-11,7)\). The new model can reproduce the first model's fit and add \(0.1z_i\) to each fitted value. The resulting fitted values are \((1,2,2,5)\), exactly the observed outcomes, so \(SSE_{x,z}=0\). Therefore,
The outcome, observations, and \(SST=9\) are unchanged; the larger model is nested because it contains the intercept and \(x\) terms. Its R-squared rises from \(0.8\) to \(1\), as the result predicts. In this deliberately constructed example, the added predictor reproduces the remaining residuals exactly. The value \(R^2=1\) describes this sample's fit, not evidence that the predictor has a reliable relationship beyond these four observations.
Adding a Predictor Can Also Leave R-Squared Unchanged
The inequality allows equality. A predictor can add no improvement if its direction contributes nothing to the residual variation that remains after fitting the smaller model. A zero estimated coefficient is one possible reason; more generally, the larger model may have additional columns without improving its best attainable fit.
Worked Example: A Predictor With No Additional Fit
Use the same four hypothetical outcomes \(y=(1,2,2,5)\) and predictor \(x=(0,1,2,3)\). The intercept-and-\(x\) fit and its residuals are as above: the fitted values are \((0.7,1.9,3.1,4.3)\), the residuals are \(e=(0.3,0.1,-1.1,0.7)\), and \(SSE_x=1.8\). Consider adding \(z=(3,-5,1,1)\).
The added predictor is centered and orthogonal to \(x\): its values sum to zero, and its products with \(x\) sum to \(0(3)+1(-5)+2(1)+3(1)=0\). It is also orthogonal to the existing residuals:
Thus, after accounting for the intercept and \(x\), the least-squares coefficient of \(z\) is zero: the residuals have zero inner product with \(z\), so adding a multiple of \(z\) cannot reduce their squared sum. The larger model has \(SSE_{x,z}=1.8\), and its R-squared is
The value is unchanged from the smaller model. This example shows why “R-squared cannot decrease” does not mean “R-squared must increase.” It can remain the same when an added predictor has no incremental least-squares contribution.
Why the Maximum Does Not Penalize Complexity
If a candidate set consists of nested models, each adding terms to the preceding model, ordinary R-squared is nondecreasing along that sequence. Consequently, simply choosing the model with the largest in-sample R-squared will select the final, largest model whenever each addition produces any improvement. If additions yield ties, the maximum may be shared by larger and smaller models. Either way, the R-squared formula contains no term that increases as the model gains predictors.
The reason is visible in the formula: for a fixed outcome and sample, maximizing \(R^2=1-SSE/SST\) is exactly the same as minimizing the training-sample \(SSE\). The calculation rewards reductions in residual sum of squares but does not subtract a cost for estimating more coefficients. Additional flexibility can capture genuine structure, but it can also fit idiosyncrasies in the observed sample. Ordinary R-squared alone does not distinguish those possibilities.
Worked Example: The Highest In-Sample R-Squared Is Not a Complexity Ranking
Return to the hypothetical data in the first worked example. Compare the intercept-only model, the intercept-and-\(x\) model, and the intercept-\(x\)-and-\(z\) model. They use the same four outcomes, so \(SST=9\) for each. For the intercept-only model, every fitted value is \(\bar{y}=2.5\), giving \(SSE=9\). The other two models had \(SSE=1.8\) and \(SSE=0\), respectively.
| Candidate model | Number of predictors | SSE | Ordinary R-squared |
|---|---|---|---|
| Intercept only | 0 | 9 | \(1-9/9=0\) |
| Intercept and \(x\) | 1 | 1.8 | \(1-1.8/9=0.8\) |
| Intercept, \(x\), and \(z\) | 2 | 0 | \(1-0/9=1\) |
By maximum ordinary R-squared, the model with both predictors wins. That ranking says only that this model fits these four outcomes at least as closely as the others. It does not establish that the second predictor is substantively important, that the model is preferable for the scientific purpose, or that its predictions will be better for new cases. A decision about model quality needs a criterion that addresses the purpose and the cost of complexity; the maximum ordinary R-squared does not do that by itself.
Use R-Squared Comparisons Carefully
Ordinary R-squared can describe how much of the observed variation around the mean is accounted for by a fitted model, in the in-sample least-squares sense. It is therefore informative as a summary of fit. Its monotonicity is also a useful diagnostic expectation: if a model truly nests another and uses the same data and outcome, a smaller ordinary R-squared for the larger model signals that some comparison condition or calculation should be checked.
A direct comparison requires care. If candidate models use different observations, their \(SST\) values can differ, and their R-squared values no longer reflect a common denominator or exactly the same data. If they use different outcomes or outcome scales, the comparison also changes its target. Even with the same data, nonnested models need not have a simple ordering from their formulas; their in-sample R-squared values can still be calculated, but the nesting theorem does not establish a general complexity-adjusted ranking between them.
When a scientific question calls for retaining a required term, choosing a functional form, or respecting a hierarchy among terms, a raw R-squared maximum should not override that rationale. Model specification and model comparison answer related but distinct questions: the fit statistic summarizes one aspect of observed fit, while a model-selection decision must also reflect the intended use and the consequences of complexity.
Common Mistakes
- Wrong reasoning: “The bigger model has more terms, so it must explain more in a meaningful scientific sense.” The correction is that least-squares fitting guarantees only a non-increase in in-sample \(SSE\), and hence a non-decrease in ordinary R-squared for nested models. It does not guarantee scientific value or improved performance on new observations.
- Wrong reasoning: “If R-squared is unchanged, the nesting result failed.” Equality is allowed. An added predictor can have a zero fitted coefficient or otherwise leave the minimized residual sum of squares unchanged.
- Wrong reasoning: “The highest R-squared identifies the best model.” It identifies the best in-sample fit by this measure among the models compared. It does not penalize the number of estimated coefficients or settle which model is most useful.
- Wrong reasoning: “R-squared values are directly comparable even when models use different cases.” The comparison result assumes the same observed outcomes. Different samples can have different means and \(SST\) values, so first establish that the candidate models are being compared on a common response and data set.
Check Your Understanding
Use the definitions and results in this tutorial to answer the following questions.
- Two intercept-containing least-squares models use the same outcomes, and the smaller model's predictors are all included in the larger model. Which direction must their ordinary R-squared values satisfy, and why?
- Can adding a predictor leave ordinary R-squared unchanged? Give a condition under which this can happen.
- Why does the largest ordinary R-squared in a nested candidate sequence fail to provide a penalty for complexity?
- What comparison conditions are needed for the monotonicity result to apply?
- What does an ordinary R-squared of \(1\) establish about a model fitted to a sample, and what does it not establish about performance beyond that sample?