Deriving the Adjusted R-Squared Penalty

Why Adjusted R-Squared Has a Penalty

Ordinary \(R^2\) summarizes the fraction of variation around the sample mean accounted for by a regression fit. When predictors are added to a least-squares model, its residual sum of squares cannot increase, so ordinary \(R^2\) cannot decrease. That makes ordinary \(R^2\) useful for describing fit, but by itself it does not account for how many predictor coefficients were estimated.

Adjusted \(R^2\) makes a specific correction: it compares residual variation per residual degree of freedom with total variation per total degree of freedom. This tutorial derives that expression and uses it to answer a practical question: when a model uses more predictors, how much must the residual sum of squares (RSS) decrease for adjusted \(R^2\) to increase?

Definition and Conditions

Consider a least-squares regression model with an intercept and \(p\) predictor coefficients, fit to the same \(n\) observations on a quantitative response. Let \(y_i\) be the observed response and \(\hat{y}_i\) its fitted value. Define the residual sum of squares as \(\mathrm{RSS}=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\), and the total sum of squares around the sample mean as \(\mathrm{TSS}=\sum_{i=1}^{n}(y_i-\bar{y})^2\).

The intercept model has \(p+1\) estimated coefficients. If the design matrix has full column rank, the residual degrees of freedom are \(n-p-1\); the total degrees of freedom around the sample mean are \(n-1\). The corresponding mean squares are the sums of squares divided by their degrees of freedom.

The ratio compares two mean squares, each measured in squared response units. Their ratio, and therefore adjusted \(R^2\), is unitless. The condition \(\mathrm{TSS}>0\) matters: if every observed response is identical, then \(\mathrm{TSS}=0\), and the denominator of the ratio is zero. Full rank and positive residual degrees of freedom alone do not prevent that problem.

This is an algebraic fit summary. The formula and the comparisons below do not require normally distributed errors or an independence assumption. Such assumptions can matter for inference about regression parameters, but they are not needed to calculate adjusted \(R^2\) from a given fit.

Deriving the Reduction Threshold

Now compare two fits to the same response observations, with an intercept in each. Suppose the first fit has \(p\) predictor coefficients and RSS \(\mathrm{RSS}_{\mathrm{old}}\). The second has \(q\) additional predictor coefficients and RSS \(\mathrm{RSS}_{\mathrm{new}}\). Write \(d=n-p-1\) for the first fit's residual degrees of freedom; the second has \(d-q\). Assume both fits have full rank, both residual degrees of freedom are positive, and \(\mathrm{TSS}>0\). Because the response and observations are unchanged, both fits have the same positive total mean square, \(\mathrm{TSS}/(n-1)\).

Adjusted \(R^2\) rises exactly when the new residual mean square is smaller than the old one. To see this, subtract each residual-mean-square ratio from one. Since the total mean square is common and positive, the comparison reduces to comparing the two residual mean squares:

Let \(\Delta\mathrm{RSS}=\mathrm{RSS}_{\mathrm{old}}-\mathrm{RSS}_{\mathrm{new}}\), the amount by which RSS falls. Substituting \(\mathrm{RSS}_{\mathrm{new}}=\mathrm{RSS}_{\mathrm{old}}-\Delta\mathrm{RSS}\) into the inequality gives

Both degrees of freedom are positive, so multiplying by \(d(d-q)\) preserves the inequality. Expanding and cancelling the shared term gives

Dividing by \(d>0\) yields the threshold. Equality means adjusted \(R^2\) is unchanged; a smaller reduction makes it fall.

The threshold grows with \(q\): adding more coefficients requires a larger total RSS reduction to compensate for the added degrees-of-freedom penalty. It is not a fixed amount for every data set. It depends on the old fit's residual mean square. The condition is an exact algebraic comparison, not a significance test and not a rule that the added predictors are practically useful.

Using the Penalty Carefully

For a fit with an intercept and \(p\) predictors, ordinary \(R^2\) uses the ratio \(\mathrm{RSS}/\mathrm{TSS}\), whereas adjusted \(R^2\) uses the ratio of mean squares. The practical comparison is therefore not simply “did RSS fall?” but “did RSS fall enough per added coefficient?” If \(q\) coefficients are added, compare the RSS reduction with \(q\) times the old residual mean square.

A common mistake is to reason that any decrease in RSS must increase adjusted \(R^2\). That conclusion confuses ordinary \(R^2\), which cannot decrease when predictors are added in a least-squares fit on the same observations, with adjusted \(R^2\), which accounts for residual degrees of freedom. For example, a reduction of three units in the second worked example was real, but the one added coefficient required a reduction greater than four units for adjusted \(R^2\) to rise.

The threshold also has limits as a decision rule. It describes how one numerical fit summary changes; it does not establish that an added predictor has a causal effect, that the model predicts well on new observations, or that the change is statistically significant. The comparison is meaningful only when the response, observations, and total sum of squares are the same for both fits and the adjusted values are defined.