Tutorials › Regression Analysis (Undergraduate) › Why Adding Predictors Cannot Lower R-Squared

Home  /  Tutorials  /  Model Fit and Nested Comparisons

Model Fit and Nested Comparisons · Tutorial 252 of 991

Why Adding Predictors Cannot Lower R-Squared

Learn why retaining the original predictors guarantees that ordinary least-squares R-squared stays the same or increases, and which comparison conditions make that guarantee valid.

Intermediate 9 min read

What You'll Learn

  • State the conditions for comparing R-squared values across models.
  • Prove that adding predictors cannot increase the least-squares residual sum of squares.
  • Connect the residual sum of squares result to nondecreasing R-squared.
  • Work through examples where R-squared increases and where it stays unchanged.
  • Recognize why replacing a predictor or changing the observations does not carry the same guarantee.

Why the Number of Predictors Matters

Ordinary least squares chooses coefficients to make the squared residuals as small as possible. If a regression model gains predictors but keeps all its original terms, it has at least the same choices it had before: it can set the new coefficients to zero and reproduce the original fit. It may also find a better fit. This simple fact explains why adding predictors cannot lower ordinary least-squares \(R^2\), provided the models use the same outcome and observations.

The guarantee is about the fit to the data used to estimate the models. It does not, by itself, say that the added predictors are useful in practice or that predictions for new observations will improve.

Definition and Conditions

Let \(y_1,\ldots,y_n\) be the observed values of an outcome for the same \(n\) observations in both models. For a fitted model, let \(\hat{y}_i\) be its fitted value for observation \(i\). Its residual sum of squares is

$$ \operatorname{RSS}=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2. $$

For models with an intercept, the usual total sum of squares is \(\operatorname{TSS}=\sum_{i=1}^{n}(y_i-\bar{y})^2\), where \(\bar{y}\) is the sample mean of the outcome. When \(\operatorname{TSS}>0\), the usual coefficient of determination is

$$ R^2=1-\frac{\operatorname{RSS}}{\operatorname{TSS}}. $$

The comparison concerns two models where the larger model includes every predictor and term in the smaller model, with one or more terms added. Both models must use ordinary least squares, the same outcome values, and exactly the same observations. For the usual intercept-based \(R^2\), both models include an intercept.

Definition: The larger model includes the smaller model if every fitted-value pattern the smaller model can produce is also possible in the larger model. In a linear regression, this holds when the larger model retains all the smaller model's terms and adds terms, such as additional predictors.

The condition is about the models' possible fitted values, not just the number of predictor names. For example, replacing an original predictor with a different one is not adding a predictor in this sense. Keeping the same observations is also essential: it keeps the outcome values, and therefore the denominator \(\operatorname{TSS}\), fixed.

The Least-Squares Argument

Ordinary least squares selects the coefficients that minimize the sum of squared residuals within the model. Suppose the smaller model uses coefficients \(\boldsymbol{\beta}\). In the larger model, set the coefficients of every added term to zero and keep \(\boldsymbol{\beta}\) unchanged. The resulting fitted values are exactly those of the smaller model, so this is one available fit in the larger model. Because least squares in the larger model chooses a fit with the smallest possible residual sum of squares, its minimum cannot exceed the residual sum of squares of that available fit.

$$ \operatorname{RSS}_{\text{larger}} \leq \operatorname{RSS}_{\text{smaller}}. $$

With the same observations and outcome, \(\operatorname{TSS}\) is the same for both models. Since it is positive, dividing the RSS inequality by \(\operatorname{TSS}\) preserves the inequality; subtracting from 1 reverses it. Therefore,

$$ 1-\frac{\operatorname{RSS}_{\text{larger}}}{\operatorname{TSS}} \geq 1-\frac{\operatorname{RSS}_{\text{smaller}}}{\operatorname{TSS}}, \qquad R^2_{\text{larger}}\geq R^2_{\text{smaller}}. $$

The result allows equality. An added predictor might not provide a fit that improves on the smaller model, so the minimum RSS can stay the same. The argument does not require that the added predictor be statistically significant or that its coefficient be nonzero.

Worked Example: Adding a Predictor Increases R-Squared

Suppose four hypothetical observations have outcome values \(y=(2,3,5,6)\) and a predictor \(x=(-1.5,-0.5,0.5,1.5)\). Compare an intercept-only model with a least-squares model that includes \(x\) and an intercept.

Intercept-only model. The fitted value for every observation is the outcome mean, \(\bar{y}=(2+3+5+6)/4=4\). Thus \(\operatorname{RSS}=(-2)^2+(-1)^2+1^2+2^2=10\). The total sum of squares is also \(10\), so \(R^2=1-10/10=0\).

Model with \(x\). The predictor has mean zero. Its least-squares slope is

$$ \hat{\beta}_1 = \frac{\sum_{i=1}^{4}(x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^{4}(x_i-\bar{x})^2} = \frac{7}{5} = 1.4. $$

The intercept is \(4\), so the fitted values are \(4+1.4x_i=(1.9,3.3,4.7,6.1)\). The residuals are \((0.1,-0.3,0.3,-0.1)\), giving

$$ \operatorname{RSS} = 0.1^2+(-0.3)^2+0.3^2+(-0.1)^2 = 0.20, \qquad R^2=1-\frac{0.20}{10}=0.98. $$

The models use the same four outcomes, and the second retains the intercept-only model's fitted values as a possible fit by setting the slope to zero. Here the best fit with \(x\) has smaller RSS, so \(R^2\) rises from \(0\) to \(0.98\).

Worked Example: Adding a Predictor Leaves R-Squared Unchanged

Consider hypothetical outcomes \(y=(9,7,11,13)\) and two centered predictors \(x=(-1,-1,1,1)\) and \(z=(-1,1,-1,1)\). Compare the model with an intercept and \(x\) to the model that also includes \(z\).

The mean outcome is \(10\). In the model with \(x\), the fitted values are \((8,8,12,12)\); the residuals are \((1,-1,-1,1)\). The outcome deviations from its mean are \((-1,-3,1,3)\), so \(\operatorname{TSS}=1+9+1+9=20\). Consequently, the smaller model has \(\operatorname{RSS}=4\) and \(R^2=1-4/20=0.80\).

To check whether \(z\) improves the fit, note that it is orthogonal to both the intercept and \(x\): its entries sum to zero, and \(\sum x_i z_i=1-1-1+1=0\). It is also orthogonal to the smaller model's residuals:

$$ \sum_{i=1}^{4} z_i(y_i-\hat{y}_i) = (-1)(1)+(1)(-1)+(-1)(-1)+(1)(1) = 0. $$

Thus the least-squares coefficient of \(z\) in the expanded model is zero; including \(z\) does not change the fitted values. The expanded model still has \(\operatorname{RSS}=4\) and \(R^2=0.80\). This is equality in the guarantee, not a contradiction: the larger model can match the original fit but has no better fit for these data.

When the Guarantee Does Not Apply

The conditions matter. If a predictor is replaced rather than added while retaining the original terms, the new model may not be able to reproduce the old model's fit. If the observations differ, the models are fitting different outcome values and have different sums of squares to compare. In either case, the argument that the larger model can copy the smaller model's fit is unavailable.

Worked Example: Replacing a Predictor Can Lower R-Squared

Use the same hypothetical outcomes \(y=(9,7,11,13)\) from the previous example. Compare the intercept-and-\(x\) model with an intercept-and-\(z\) model, where \(x=(-1,-1,1,1)\) and \(z=(-1,1,-1,1)\). This time, the second model replaces \(x\); it does not retain \(x\).

As calculated above, the intercept-and-\(x\) model has \(\operatorname{RSS}=4\), \(\operatorname{TSS}=20\), and \(R^2=0.80\). For the intercept-and-\(z\) model, the least-squares slope is zero because

$$ \sum_{i=1}^{4} z_i(y_i-\bar{y}) = (-1)(-1)+(1)(-3)+(-1)(1)+(1)(3) = 0. $$

Its fitted values are therefore all \(10\), and its residuals are \((-1,-3,1,3)\). Its RSS is \(1+9+1+9=20\), so \(R^2=1-20/20=0\). The decrease from \(0.80\) to \(0\) is possible because the \(z\)-only model cannot reproduce the fit from the \(x\)-only model. The two models are not related by adding \(z\) while keeping \(x\).

Interpreting the Result Carefully

The guarantee is a mathematical property of minimizing squared residuals, not evidence that every added predictor is informative. A predictor can leave \(R^2\) unchanged, as in the second example, or improve it by a very small amount. Even a substantial increase describes improved fit to the observations used to estimate the model; it does not establish a causal effect or guarantee better predictions on new observations.

A common mistake is to compare two models with different predictors or different samples and explain any change in \(R^2\) by saying that “more predictors always increase \(R^2\).” That statement is too broad. The correct reasoning is: when a least-squares model retains every term of another model, adds terms, and uses the same outcome observations and the same \(R^2\) definition, its \(R^2\) cannot be smaller. If any of those conditions changes, check the comparison rather than assuming the guarantee applies.

Key takeaway: Added terms give ordinary least squares more possible fits, including every fit available before. Therefore RSS cannot increase; with a fixed, positive total sum of squares, \(R^2\) cannot decrease.

Check Your Understanding

Use the fixed-sample, ordinary least-squares conditions from this tutorial.

  1. Why can the larger model's minimum RSS not exceed the smaller model's RSS?
  2. What must stay fixed for the usual \(R^2\) denominator to be the same in both models?
  3. If a newly added predictor has a least-squares coefficient of zero, what can happen to \(R^2\)?
  4. Does the nondecrease guarantee apply when one predictor is removed and a different one is substituted? Explain.
  5. Does a higher in-sample \(R^2\) alone guarantee better predictions for new observations? Explain briefly.

Previous

Next

Back to Tutorials