Comparing Models on the Same Outcome and Sample

Why the Sample Must Stay Fixed

A regression model can look better simply because it was evaluated on an easier or smaller set of observations. A model’s residual sum of squares (RSS) is the sum of its squared prediction errors on the observations used to fit it. If two models are fitted to the same outcome values for the same individuals, their RSS values describe fit on a shared basis. If they use different observations, they do not.

Adjusted R-squared also depends on the response values in the sample, as well as on the model’s residual degrees of freedom. It can help compare models of different sizes when the outcome observations are identical, but it does not make comparisons across different samples automatically fair. The comparisons here concern observed, in-sample fit; neither statistic alone establishes that a model is useful for another purpose.

Set Up a Fair Fit Comparison

Suppose each candidate model is an ordinary least-squares regression with an intercept and the same quantitative response $y$, recorded for the same $n$ observations. The predictor sets may differ. For each model, fitted values are obtained by least squares, and the residual for observation $i$ is the observed response minus its fitted value. RSS is the sum of these residuals squared. Define the total sum of squares, SST, as the sum of squared deviations of the observed responses from their sample mean.

Definition: For a model fitted to $n$ response observations, let $r$ be the rank of its design matrix, including the intercept column. Its residual degrees of freedom are $n-r$. With an intercept, define $$ \mathrm{RSS}=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2, \qquad \mathrm{SST}=\sum_{i=1}^{n}(y_i-\bar{y})^2, \qquad R^2_{\mathrm{adj}}=1-\frac{\mathrm{RSS}/(n-r)}{\mathrm{SST}/(n-1)}. $$ Here $\hat{y}_i$ is the fitted response and $\bar{y}$ is the sample mean of the response values. If there are $k$ predictor columns and the intercept and all $k$ columns are linearly independent, then $r=k+1$, so $n-r=n-k-1$. In general, use the matrix rank $r$, not a count of predictor terms, to find the residual degrees of freedom.

These fit summaries require a common response measured on the same scale, identical response observations, and least-squares fits with an intercept. For the adjusted R-squared formula, the model must have positive residual degrees of freedom and the sample response must have positive SST. Rank accounts for linearly dependent columns: they do not add an independent dimension to the fitted model. These are descriptive fit calculations; normality or a significance test is not required to compute them.

For two models on the same observations, SST and $n-1$ are fixed. Therefore, the model with smaller RSS has larger ordinary $R^2=1-\mathrm{RSS}/\mathrm{SST}$. Adjusted R-squared also accounts for residual degrees of freedom, so it may rank the models differently. Neither statistic is a formal test of whether a predictor has an effect.

Worked Comparisons on the Same Observations

Worked Example: Comparing RSS and Adjusted R-Squared

Suppose two hypothetical models are fitted to the same 10 patients’ outcome measurements, recorded in the same units. Model A has one predictor, full design-matrix rank $r_A=2$, and RSS $=300$ outcome-units squared. Model B adds a predictor, has rank $r_B=3$, and RSS $=220$ outcome-units squared. For these observations, SST $=1000$ outcome-units squared. Compare their fit summaries.

The response observations and SST are shared, so the RSS values can be compared directly: $220<300$. Model B has smaller in-sample squared residual error. Its ordinary R-squared is also larger:

R^2_A=1-\frac{300}{1000}=0.700, \qquad R^2_B=1-\frac{220}{1000}=0.780.

For adjusted R-squared, Model A has $n-r_A=10-2=8$ residual degrees of freedom, and Model B has $10-3=7$. The common response variance denominator is $\mathrm{SST}/(n-1)=1000/9=111.111$:

R^2_{\mathrm{adj},A} =1-\frac{300/8}{1000/9} =1-\frac{37.5}{111.111} =0.6625, \qquad R^2_{\mathrm{adj},B} =1-\frac{220/7}{1000/9} \approx 0.7171.

On these data, Model B has both smaller RSS and larger adjusted R-squared. The conclusion is limited to the observed fit summaries: it does not establish that Model B is preferable for every purpose.

Worked Example: A Smaller RSS Can Come With Lower Adjusted R-Squared

In a second hypothetical comparison, both models use the same 12 observations and the same outcome units. SST is $120$ outcome-units squared. Model A has rank $r_A=2$ and RSS $=60$; Model B has rank $r_B=4$ and RSS $=58$. Model B fits the observed data with slightly smaller RSS, but the increase in rank also changes the adjusted calculation.

The ordinary R-squared values are:

R^2_A=1-\frac{60}{120}=0.500, \qquad R^2_B=1-\frac{58}{120}\approx 0.5167.

The response variance denominator for adjusted R-squared is $120/(12-1)=120/11\approx10.9091$. Model A has $12-2=10$ residual degrees of freedom, while Model B has $12-4=8$:

R^2_{\mathrm{adj},A} =1-\frac{60/10}{120/11} =1-\frac{6}{10.9091} =0.450, \qquad R^2_{\mathrm{adj},B} =1-\frac{58/8}{120/11} =1-\frac{7.25}{10.9091} \approx 0.3354.

Thus, Model B has the smaller RSS and the larger ordinary R-squared, but Model A has the larger adjusted R-squared. The metrics answer different descriptive questions: ordinary R-squared records the reduction in residual variation relative to SST, while adjusted R-squared also uses the model’s residual degrees of freedom. This comparison does not itself decide which model to use.

Why Rank Matters in Adjusted R-Squared

Counting predictor terms gives the correct residual degrees of freedom only when the intercept and predictor columns are linearly independent. If some columns are exact linear combinations of others, the model has fewer independent fitted dimensions than its number of terms suggests. The rank-based definition handles this directly.

Worked Example: Two Terms but Only One Independent Predictor Column

Consider a hypothetical sample of 10 observations with an intercept and two recorded predictor columns, where the second column is exactly twice the first. Assume the first predictor varies across observations. The two predictor columns are linearly dependent, so the design matrix has rank $r=2$: one dimension for the intercept and one for the varying predictor. Suppose SST is $90$ outcome-units squared and RSS is $30$ outcome-units squared.

The correct residual degrees of freedom are $n-r=10-2=8$, and $\mathrm{SST}/(n-1)=90/9=10$. Therefore:

R^2_{\mathrm{adj}} =1-\frac{30/8}{90/9} =1-\frac{3.75}{10} =0.625.

Using the count of two predictor terms as if they were independent would incorrectly give $10-2-1=7$ residual degrees of freedom and $1-(30/7)/10\approx0.5714$. The predictor columns do not provide two independent directions for fitting, so that calculation is not appropriate. Coefficients for the two redundant columns are not separately determined, but the rank still describes the dimension of the fitted model.

Why Different Samples Obstruct Direct Comparison

When models use different response observations, their RSS values add squared errors over different sets of cases. A smaller RSS may simply reflect fewer observations. Adjusted R-squared does not solve this problem: SST, sample size, and response variability can all change when the cases change. A difference in either fit statistic may therefore combine a model difference with a sample difference.

Worked Example: A Smaller RSS on a Smaller Sample

Suppose two hypothetical fits use the same outcome and units but not the same observations. Model A uses 8 cases, has rank $2$, SST $=672$ outcome-units squared, and RSS $=40$. Model B uses a subset of 5 cases, has rank $2$, SST $=40$ outcome-units squared, and RSS $=20$. Both RSS values are possible for least-squares fits with an intercept because each is below its model’s SST.

The raw RSS is smaller for Model B ($20<40$), but it is calculated over only 5 observations rather than 8. Their adjusted R-squared values also differ:

R^2_{\mathrm{adj},A} =1-\frac{40/(8-2)}{672/(8-1)} =1-\frac{6.6667}{96} \approx 0.9306,

R^2_{\mathrm{adj},B} =1-\frac{20/(5-2)}{40/(5-1)} =1-\frac{6.6667}{10} =0.3333.

These values do not provide a direct ranking of the models, because they describe fit on different response samples with different SST and sample size. The arithmetic is valid for each sample individually; the problem is interpreting their difference as a model-only comparison. A direct comparison requires refitting both models to the same response observations.

A Useful Adjusted R-Squared Comparison Rule

The adjusted R-squared formula also gives a simple way to understand its penalty. On a fixed sample with positive SST, a model has higher adjusted R-squared exactly when its residual mean square, $\mathrm{RSS}/(n-r)$, is smaller. This follows because the common denominator $\mathrm{SST}/(n-1)$ is positive and fixed.

For example, suppose a larger model adds $q$ independent dimensions to a smaller model, so its residual degrees of freedom fall from $\nu$ to $\nu-q$. If RSS falls by $\Delta$, then the larger model has higher adjusted R-squared precisely when:

\frac{\mathrm{RSS}_{\mathrm{old}}-\Delta}{\nu-q} < \frac{\mathrm{RSS}_{\mathrm{old}}}{\nu}.

Multiplying both sides by the positive quantity $\nu(\nu-q)$ and cancelling $\nu\mathrm{RSS}_{\mathrm{old}}$ gives:

\nu(\mathrm{RSS}_{\mathrm{old}}-\Delta) < (\nu-q)\mathrm{RSS}_{\mathrm{old}} \quad\Longleftrightarrow\quad \nu\Delta>q\mathrm{RSS}_{\mathrm{old}} \quad\Longleftrightarrow\quad \frac{\Delta}{q}> \frac{\mathrm{RSS}_{\mathrm{old}}}{\nu}.

So the reduction in RSS per added dimension must exceed the smaller model’s residual mean square for adjusted R-squared to increase. This is an algebraic comparison rule, not a significance test.

Common Mistakes and Practical Use

Comparing raw RSS across unequal sample sizes. A smaller sample often has fewer squared residuals to add. Compare models on the same response observations instead.
Counting terms instead of using rank. The shortcut $n-k-1$ requires the intercept and all $k$ predictor columns to be linearly independent. Otherwise use $n-r$.
Treating adjusted R-squared as a formal test. A higher value is a descriptive fit comparison, not a p-value or evidence by itself that a difference is statistically significant.
Assuming a shared sample whenever the outcome name matches. Check the actual observations used. Missing outcomes or predictors can cause models to be fitted to different cases.
Ignoring the outcome scale. RSS is in squared outcome units. It is only directly meaningful as a comparison when the response and its measurement scale are the same.

Key takeaway: For a direct fit comparison, hold the response observations and scale fixed. RSS then compares total squared residual error; adjusted R-squared also accounts for model rank through residual degrees of freedom. If samples differ, neither statistic isolates the model difference.

Check Your Understanding

Answer each question using the definitions and comparison conditions above.

Two models use the same 20 response observations and scale. What quantity is fixed in their adjusted R-squared calculations, and why does that matter?
A model has 3 predictor columns, but one is an exact linear combination of the other two. What information is needed to determine its residual degrees of freedom?
On the same sample, a larger model has lower RSS but lower adjusted R-squared. Is that possible? Explain what the two results indicate.
Why is a smaller RSS from a model fitted to 6 cases not directly comparable with an RSS from a model fitted to 10 cases?
Does a higher adjusted R-squared establish that a model is statistically significant? Explain briefly.