Adjusted R-Squared and Its Penalty

Why Adjusted R-Squared Is Needed

Ordinary $R^2$ summarizes how much of the observed variation in an outcome is accounted for by a least-squares model. When a predictor is added to a model fitted to the same observations, ordinary $R^2$ cannot decrease: the expanded model can always reproduce the old fit by assigning the new coefficient a value of zero. This means that $R^2$ alone tends to reward larger models, even when their extra predictors improve the fit only slightly.

Adjusted $R^2$ modifies this summary by accounting for how many regression coefficients were estimated. It is useful when comparing candidate models fitted to the same outcome on the same observations. Its adjustment is a degrees-of-freedom penalty: a more complex model must reduce its residual variation enough to compensate for having fewer residual degrees of freedom.

Definition: For a least-squares regression with an intercept, let $n$ be the number of observations, $k$ the number of fitted slope coefficients (not counting the intercept), $SSE$ the residual sum of squares, and $SST$ the total sum of squares about the sample mean of the outcome. Provided $n-k-1>0$ and $SST>0$, the adjusted coefficient of determination is $$ R^2_{\mathrm{adj}}=1-\frac{SSE/(n-k-1)}{SST/(n-1)} =1-(1-R^2)\frac{n-1}{n-k-1}, $$ where $R^2=1-SSE/SST$. The quantities $n-k-1$ and $n-1$ are the residual and total degrees of freedom, respectively.

The formula compares two mean squares: the residual mean square $SSE/(n-k-1)$ and the total mean square $SST/(n-1)$. The penalty is visible in the factor $(n-1)/(n-k-1)$, which is greater than 1 whenever $k>0$. Thus, for a model with at least one slope coefficient, adjusted $R^2$ is no greater than ordinary $R^2$. If $k=0$, the intercept-only model has $SSE=SST$, and both values are 0.

The definition applies to the usual intercept-included least-squares setting. It does not require normally distributed errors to calculate or compare the statistic. But a fair comparison does require the models to use the same outcome values and the same observations; otherwise, the sums of squares and their denominators do not refer to the same data. Count fitted slope coefficients in $k$, including separate coefficients used to represent a categorical predictor.

How the Penalty Changes a Model Comparison

For a fixed data set and outcome, $n$ and $SST$ stay the same across candidate models. Adding coefficients lowers the residual degrees of freedom. A model with more coefficients therefore needs a sufficiently smaller $SSE$ to lower its residual mean square $SSE/(n-k-1)$. Adjusted $R^2$ rewards a reduction in residual variation only when it is large enough to offset this change.

Worked Example: Comparing Two Models With the Same Outcome

Suppose two hypothetical least-squares models with intercepts are fitted to the same 30 observations of an outcome measured in points. Model A has $k=2$ slope coefficients and $SSE=360$ points-squared. Model B has $k=4$ slope coefficients and $SSE=300$ points-squared. For both models, $SST=900$ points-squared. The models use the same outcome and observations, so adjusted $R^2$ is suitable for this comparison. Both residual degrees of freedom are positive.

For Model A, the residual degrees of freedom are $30-2-1=27$. Its ordinary $R^2$ and adjusted $R^2$ are $$ R^2_A=1-\frac{360}{900}=0.6000, \qquad R^2_{\mathrm{adj},A} =1-\frac{360/27}{900/29} =1-\frac{10440}{24300} \approx 0.5704. $$

For Model B, the residual degrees of freedom are $30-4-1=25$. Its corresponding values are $$ R^2_B=1-\frac{300}{900}=0.6667, \qquad R^2_{\mathrm{adj},B} =1-\frac{300/25}{900/29} =1-\frac{8700}{22500} \approx 0.6133. $$

Model B has the higher adjusted $R^2$, about 0.6133 versus 0.5704. Its smaller residual sum of squares more than compensates for the reduction in residual degrees of freedom. This ranking says which model has the higher adjusted $R^2$ on these data; by itself, it does not establish that Model B will predict better on new observations.

When Does an Added Predictor Raise Adjusted R-Squared?

The penalty can be made precise when one coefficient is added to a model and the outcome and observations remain unchanged. Write $SSE_{\mathrm{old}}$ for the residual sum of squares before adding the coefficient and $SSE_{\mathrm{new}}$ for the value afterward. Let the new model have $d$ residual degrees of freedom; the old model then has $d+1$. The total mean square $SST/(n-1)$ is common to both models. Adjusted $R^2$ rises exactly when the new model's residual mean square is lower:

\frac{SSE_{\mathrm{new}}}{d} < \frac{SSE_{\mathrm{old}}}{d+1} \quad\Longleftrightarrow\quad d\bigl(SSE_{\mathrm{old}}-SSE_{\mathrm{new}}\bigr)>SSE_{\mathrm{new}}.

Rearranging the right-hand inequality gives $$ \frac{SSE_{\mathrm{old}}-SSE_{\mathrm{new}}}{SSE_{\mathrm{new}}/d}>1. $$ For a single added coefficient, the left side is the partial $F$ statistic for testing whether that coefficient is zero, when the usual least-squares partial $F$ statistic is defined. This algebraic connection does not turn adjusted $R^2$ into a significance test: its increase rule corresponds to a cutoff of 1, not to a chosen significance level. The calculation explains the penalty rather than supplying a general hypothesis-testing procedure.

Worked Example: A Small Fit Improvement Is Not Enough

Consider a hypothetical outcome measured on 20 observations. Both models include an intercept and use the same observations; the old model has $k=2$ slope coefficients, while the new model adds one coefficient, so $k=3$. Suppose $SST=190$ points-squared, the old model has $SSE_{\mathrm{old}}=100$ points-squared, and the new model has $SSE_{\mathrm{new}}=96$ points-squared. The residual degrees of freedom are 17 for the old model and 16 for the new one, both positive.

Ordinary $R^2$ rises because $SSE$ falls: $$ R^2_{\mathrm{old}}=1-\frac{100}{190}\approx 0.4737, \qquad R^2_{\mathrm{new}}=1-\frac{96}{190}\approx 0.4947. $$

But the adjusted values are $$ R^2_{\mathrm{adj,old}} =1-\frac{100/17}{190/19} =1-\frac{1900}{3230} \approx 0.4118, $$ $$ R^2_{\mathrm{adj,new}} =1-\frac{96/16}{190/19} =1-\frac{6}{10} =0.4000. $$

The added coefficient reduces the residual sum of squares by only $100-96=4$ points-squared. The new model's residual mean square is $96/16=6$, while the old model's is $100/17\approx 5.882$. Because the residual mean square increases, adjusted $R^2$ decreases, even though ordinary $R^2$ increases. The comparison indicates that this small fit improvement does not offset the degrees-of-freedom penalty.

Reading the Value and Comparing Models Carefully

Adjusted $R^2$ is a scaled fit summary, not a literal proportion of sample variation explained in the same direct sense as ordinary $R^2$. It can be negative: that occurs when the residual mean square exceeds the total mean square. A negative value is possible even with an intercept, and it does not imply that the arithmetic is invalid. It indicates that, after the adjustment, the model's residual variation is large relative to the outcome's total variation.

Use adjusted $R^2$ as a compact comparison criterion, not as a stand-alone decision rule. A larger value does not verify that the candidate predictors are scientifically justified, that the model form is appropriate, or that forecasts will be accurate beyond the fitted data. Those questions depend on the purpose of the analysis and on evidence beyond this one statistic. The comparison also becomes inappropriate if candidate models use different observations—for example, because missing values caused one model to be fitted to a smaller subset.

Worked Example: A More Complex Model Can Rank Lower

For a final hypothetical comparison, suppose two intercept-included models use the same 30 observations and outcome, measured in points, with $SST=900$ points-squared. Model D has $k=3$ slope coefficients and $SSE=360$ points-squared. Model C has $k=6$ slope coefficients and $SSE=320$ points-squared. Their residual degrees of freedom are 26 and 23, respectively; both are positive, and the common data make the comparison meaningful.

Model D has $$ R^2_D=1-\frac{360}{900}=0.6000, \qquad R^2_{\mathrm{adj},D} =1-\frac{360/26}{900/29} =1-\frac{10440}{23400} \approx 0.5538. $$

Model C has $$ R^2_C=1-\frac{320}{900}\approx 0.6444, \qquad R^2_{\mathrm{adj},C} =1-\frac{320/23}{900/29} =1-\frac{9280}{20700} \approx 0.5517. $$

Ordinary $R^2$ is higher for Model C, but adjusted $R^2$ is slightly lower: about 0.5517 rather than 0.5538. The calculation reflects the larger model's greater penalty. The difference is small, so it should not be portrayed as decisive evidence that Model D is scientifically or predictively superior.

Common Mistakes

Assuming adjusted $R^2$ must rise when a predictor is added. Ordinary $R^2$ cannot fall in a nested least-squares model, but adjusted $R^2$ can fall if the reduction in $SSE$ is too small.
Forgetting the intercept in the degrees of freedom. With $k$ slope coefficients and an intercept, residual degrees of freedom are $n-k-1$, not $n-k$.
Comparing models fitted to different observations. The adjusted values then use different underlying outcome data and are not a clean model comparison.
Calling adjusted $R^2$ the probability that a model is correct. It is a descriptive fit statistic, not a probability, a hypothesis test, or a guarantee of performance on new data.
Rounding too early. Keep the sums of squares and degree-of-freedom fractions unrounded through the calculation, then round the final statistic consistently.

Key takeaway: Adjusted $R^2$ compares residual variation per residual degree of freedom with total variation per total degree of freedom. Use it to rank least-squares models only when they have the same outcome and observations, and interpret its ranking alongside the purpose and specification of the models.

Check Your Understanding

Try these questions using the definition and comparisons above.

A model has $n=40$ observations and $k=5$ slope coefficients. What are its residual degrees of freedom when it includes an intercept?
Why can ordinary $R^2$ increase while adjusted $R^2$ decreases after adding a predictor?
Two candidate models use different sets of observations because of missing values. Why is their adjusted $R^2$ comparison not directly fair?
Can adjusted $R^2$ be negative for an intercept-included model? What does that indicate about its residual mean square relative to its total mean square?
Does a higher adjusted $R^2$ prove that a model will predict more accurately on new observations? Explain briefly.