Limits Of Adjusted R-Squared

What Adjusted R-Squared Can—and Cannot—Tell You

Adjusted $R^2$ is useful when comparing regression models fitted to the same response on the same observations. Unlike ordinary $R^2$, it accounts for the number of predictors by comparing residual variation per residual degree of freedom. That adjustment is a useful penalty, but it is not a certificate: the model with the largest adjusted $R^2$ need not represent the correct relationship, support valid post-selection inference, or predict best on new data.

These distinctions matter in model selection. A score is calculated from the observed sample; model specification concerns whether the chosen regression adequately represents the underlying process; inference concerns the reliability of conclusions drawn from the data; prediction concerns performance on observations not used to fit the model. A favorable value for one quantity does not settle the others.

Definition: For a linear regression with an intercept, $n$ observations, $p$ predictors (not counting the intercept), residual sum of squares $RSS$, and total sum of squares $TSS=\sum_{i=1}^n(y_i-\bar y)^2$, define $$ R^2_{\mathrm{adj}}=1-\frac{RSS/(n-p-1)}{TSS/(n-1)}. $$ This quantity is defined when $n-p-1>0$ and $TSS>0$. For comparisons, all models must use the same response observations, so $TSS$ and $n$ are common.

The formula does not require normally distributed errors simply to calculate the score. Its familiar inferential interpretations, however, require additional assumptions. For example, the usual partial $F$-test reference distribution relies on a correctly specified linear model with independent, mean-zero, normally distributed errors having a common variance, and on the models being fixed in advance. When candidate models are selected after inspecting the same data, treating the winning model as if it had been specified in advance is generally unjustified.

Why an Added Predictor Can Raise the Score

Suppose a smaller model has $p$ predictors and a larger, nested model adds one predictor. Both include an intercept, use the same $n$ observations, and have full column rank. Assume both residual degrees of freedom are positive and $TSS>0$. Write $RSS_0$ for the smaller-model residual sum of squares, $RSS_1$ for the larger-model residual sum of squares, and $\Delta=RSS_0-RSS_1$. Nested least squares ensures $\Delta\geq 0$.

When $RSS_1>0$, the exact comparison can be expressed using the partial $F$-statistic for this one added predictor. The algebra explains the threshold; it does not establish that the predictor is substantively appropriate or that the statistical assumptions for an inferential test hold.

Write the two adjusted scores.
The smaller and larger models have residual degrees of freedom $n-p-1$ and $n-p-2$, respectively. Since $TSS/(n-1)$ is the same positive quantity in both, the larger score is higher exactly when $RSS_1/(n-p-2)<RSS_0/(n-p-1)$.

Substitute the reduction in residual sum of squares.
Set $d_1=n-p-2$, so the smaller model has $d_1+1$ residual degrees of freedom and $RSS_0=RSS_1+\Delta$. The inequality becomes $RSS_1/d_1<(RSS_1+\Delta)/(d_1+1)$.

Simplify the inequality.
Multiplying by the positive $d_1(d_1+1)$ gives $RSS_1(d_1+1)<d_1(RSS_1+\Delta)$, which reduces to $RSS_1<d_1\Delta$. Thus the score rises exactly when $\Delta/(RSS_1/d_1)>1$.

The fraction in the last step is the partial $F$-statistic for one added predictor, provided $RSS_1>0$. Therefore, under these comparison conditions, adjusted $R^2$ rises exactly when that statistic exceeds 1. This is a numerical threshold for the score—not the usual threshold for statistical significance.

There is an important boundary case. If $RSS_1=0$ and $RSS_0>0$, the larger model fits the observed response exactly, so its adjusted $R^2$ is 1 and the smaller model's score is lower. The displayed fraction is undefined in this case, so it must not be used to make the comparison. If $RSS_0=0$, nesting implies $RSS_1=0$ as well; both adjusted scores equal 1, provided their residual degrees of freedom are positive and $TSS>0$.

Worked Example: A Higher Score for a Noise Predictor

A hypothetical insurer models a portfolio's annual claim cost using $n=30$ policy groups. A baseline model has one predictor; a candidate adds a second predictor that, in the hypothetical data-generating process, has no genuine relationship with claim cost. On the observed sample, let $TSS=100$, $RSS_0=80$, and $RSS_1=74$. Determine whether adjusted $R^2$ rises and what that result establishes.

Method. Compare both adjusted scores using the common $TSS$, then check the one-predictor threshold. The example illustrates that a sample score can reward a chance association; the values are hypothetical, not industry data.

The baseline has $p=1$, so its residual degrees of freedom are $30-1-1=28$. The larger model has $p=2$, so its residual degrees of freedom are $30-2-1=27$. Thus

R^2_{\mathrm{adj},0} =1-\frac{80/28}{100/29} =1-\frac{20}{7}\cdot\frac{29}{100} =1-\frac{580}{700} =0.171428\ldots

For the larger model,

R^2_{\mathrm{adj},1} =1-\frac{74/27}{100/29} =1-\frac{74\cdot29}{27\cdot100} =1-\frac{2146}{2700} =0.205185\ldots

To check the threshold directly, $\Delta=80-74=6$, and the partial $F$-statistic is $6/(74/27)=162/74=2.189189\ldots$, which exceeds 1. Rounded only at the final step to four decimal places, the scores are 0.1714 and 0.2052, respectively. The larger model wins by adjusted $R^2$, even though the added predictor has no genuine effect in the stated hypothetical process. A score increase is not proof that the predictor captures a real relationship.

Common mistake avoided: Treating an adjusted $R^2$ increase as evidence that a predictor is truly relevant. The calculation only describes the fitted sample; it cannot distinguish a real signal from a chance improvement.

A Maximum Does Not Verify Model Specification

A model can have a high adjusted $R^2$ and still be wrong in important ways. The mean relationship might be nonlinear when the fitted model is linear; an important predictor might be omitted; an included predictor might have the wrong functional form; or the error variability might not meet assumptions needed for standard inference. Adjusted $R^2$ is computed from residual sum of squares and model size, so it does not directly test all such features.

Nor does maximizing the score guarantee that the selected model is the simplest scientifically defensible one. A search over many candidate predictors or transformations can locate a model that fits peculiarities of one sample. The complexity penalty helps compare fit and size, but it does not erase the effects of searching. A selected model can therefore be too tailored to the observed outcomes, even when its adjusted $R^2$ is the largest among the candidates considered.

Worked Example: An Exact Fit and a Score of One

A hypothetical study has $n=9$ sites, and the response has $TSS=40$. A baseline regression with one predictor has $RSS_0=16$. A larger regression with three predictors fits the nine recorded responses exactly, so $RSS_1=0$. Compare the scores and explain why the usual ratio check is unavailable.

Method. Apply the definition separately, checking the residual degrees of freedom first. The baseline has $9-1-1=7$ residual degrees of freedom, and the larger model has $9-3-1=5$, so both adjusted scores are defined.

R^2_{\mathrm{adj},0} =1-\frac{16/7}{40/8} =1-\frac{16/7}{5} =1-\frac{16}{35} =\frac{19}{35} \approx 0.5429.

For the larger model,

R^2_{\mathrm{adj},1} =1-\frac{0/5}{40/8} =1.

The larger model's adjusted score is higher. But the threshold ratio $\Delta/(RSS_1/5)$ divides by zero, so it is not a valid way to express this comparison. An exact fit to a small observed sample does not by itself show that the fitted relationship will hold at new sites; it can instead reflect an overly flexible fit.

Common mistake avoided: Using the partial-$F$ ratio when the larger model has zero residual sum of squares. Handle the zero-residual case directly from the adjusted $R^2$ definition.

Selection Changes the Meaning of Inference

Ordinary regression inference is calibrated for a specified model under its assumptions. If a predictor is chosen because it produced a favorable result among many candidates, the selected coefficient and its usual standard error are no longer being interpreted in the same way as for a predictor fixed before examining the data. The selection process preferentially retains predictors that looked unusually helpful in that sample. As a result, an ordinary confidence interval or test calculated afterward may fail to account for the search and can overstate the strength of evidence.

This issue is distinct from whether the adjusted score was computed correctly. A comparison can be arithmetically correct and still be followed by misleading inference. One must report how the model was selected and avoid treating a data-selected model as if it were prespecified. The score alone does not provide a correction for this selection effect.

Worked Example: The Winner Among Many Candidate Predictors

A hypothetical portfolio analysis has $n=24$ observations. A prespecified baseline model has two predictors, $TSS=200$, and $RSS_0=120$. An analyst searches over 12 possible additional predictors and retains the one giving the best adjusted $R^2$. For the selected model, $RSS_1=105$. Compare the scores and explain what can be concluded about an ordinary test of the selected predictor.

Method. First calculate the score comparison as though comparing these two fitted models. Then account for the fact that the added predictor was chosen after a search across 12 candidates.

The baseline has $24-2-1=21$ residual degrees of freedom. The selected model has three predictors and $24-3-1=20$ residual degrees of freedom. Its adjusted score is

R^2_{\mathrm{adj},0} =1-\frac{120/21}{200/23} =1-\frac{120\cdot23}{21\cdot200} =1-\frac{23}{35} =\frac{12}{35} \approx 0.3429.

R^2_{\mathrm{adj},1} =1-\frac{105/20}{200/23} =1-\frac{105\cdot23}{20\cdot200} =1-\frac{483}{800} =\frac{317}{800} =0.39625.

Here $\Delta=120-105=15$, and the one-predictor partial statistic is $15/(105/20)=300/105=20/7\approx2.857$, which exceeds 1. Rounded to four decimal places, the baseline score is 0.3429 and the selected model's score is 0.3963. The score has increased, but the added variable was selected from 12 candidates using the same observations. An ordinary test that treats this variable as prespecified does not account for that search. The score does not establish that the selected coefficient is significant after selection, nor that the association is real.

Common mistake avoided: Calculating a conventional test for the winning predictor and ignoring that it was the winner of a search. Selection history matters even when the displayed adjusted $R^2$ arithmetic is correct.

In-Sample Fit Is Not New-Data Performance

Adjusted $R^2$ is calculated using the data that fit the model. Prediction for new observations is a different objective: it concerns how well the fitted relationship performs on observations not used to choose and estimate it. A model can improve its adjusted score in the training sample yet fail to improve, or even worsen, its predictive accuracy on new data. This can occur when the added predictor captures accidental sample variation rather than a stable relationship.

Consequently, adjusted $R^2$ is best treated as one descriptive model-comparison measure, not as a universal selection rule. Interpret it alongside the modeling purpose, the plausibility of the specification, the selection process, and appropriate assessment of prediction on data not used for fitting. A larger value is not a guarantee of better future predictions.

Common Traps

Confusing a threshold with significance. For one added predictor and positive larger-model $RSS$, adjusted $R^2$ rises when the partial $F$-statistic exceeds 1. That is not the same as meeting a conventional significance threshold.
Dividing by zero in an exact fit. If $RSS_1=0$, use the adjusted $R^2$ definition directly. The ratio $\Delta/(RSS_1/df_1)$ is undefined.
Comparing models on different observations. Different analysis samples generally have different $TSS$, so the direct comparison and derivation above do not apply as stated.
Calling the selected model correct because its score is largest. The score does not rule out omitted variables, an unsuitable functional form, or other specification problems.
Ignoring how candidates were selected. Inference after searching the same data needs to account for that selection; standard prespecified-model interpretations cannot simply be assumed.
Equating fit with prediction. An in-sample score is not a guarantee of performance on new observations.

Key takeaway: Adjusted $R^2$ compares residual variation per residual degree of freedom. A maximum is an in-sample ranking, not proof of correct specification, unbiased post-selection inference, or superior prediction on new data. These limitations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For numerical comparisons, assume the models use the same observations, include an intercept, and have positive residual degrees of freedom unless the question states otherwise.

A model with $n=18$, one predictor, $TSS=90$, and $RSS_0=60$ is expanded by one predictor, giving $RSS_1=55$. Find the larger model's residual degrees of freedom and the partial $F$-statistic. Does adjusted $R^2$ rise? (Answer: 15 degrees of freedom; $F=5/(55/15)=15/11\approx1.3636$; yes.)
A baseline model has $n=12$, two predictors, and $RSS_0=50$. Adding one predictor gives $RSS_1=48$. Calculate the partial $F$-statistic and decide whether adjusted $R^2$ rises. (Answer: Larger-model residual degrees of freedom $=12-3-1=8$; $F=2/(48/8)=1/3$; no.)
For $n=9$, $TSS=40$, and a one-predictor model with $RSS_0=16$, a nested three-predictor model has $RSS_1=0$. What is the larger model's adjusted $R^2$, and is the ratio threshold defined? (Answer: $R^2_{\mathrm{adj}}=1$; the ratio threshold is undefined because its denominator is zero.)
An analyst compares 20 candidate predictors and reports a conventional test for the one that gave the greatest adjusted $R^2$, treating it as prespecified. What inferential issue has been overlooked? (Answer: The reported test does not account for data-driven selection among 20 candidates and may overstate the evidence.)
A model has the largest adjusted $R^2$ among those fitted to a training sample. Does this establish that it will predict better on new observations or that its specification is correct? (Answer: No; adjusted $R^2$ is an in-sample score and guarantees neither correct specification nor superior new-data prediction.)

What You'll Learn