R-Squared and Adjusted R-Squared in Multiple Regression

What R-Squared Describes

A multiple regression model can summarize how a response varies across the observations used to fit it. R-squared is one measure of that summary: it compares the variation left in the residuals with the response’s total variation around its sample mean. Because adding predictors can improve this in-sample fit even when the added predictors contribute little, adjusted R-squared also accounts for the number of fitted slope terms.

Both measures describe fit for a particular response, set of observations, and fitted model. They do not establish that predictors cause changes in the response, and neither measure alone determines whether a model is useful for prediction.

Here \(k\) counts the slope coefficients, not the intercept. It counts each coefficient used for a quantitative predictor or indicator term; for a categorical predictor represented by several indicators, each indicator coefficient is a slope term. The adjustment formula needs positive residual degrees of freedom, so it is undefined if \(n-k-1=0\). The ordinary \(R^2\) calculation requires \(SST>0\), meaning the observed response values are not all identical.

The sums of squares use squared response units, so their ratios and both R-squared measures have no units. The calculations themselves do not require a normality assumption. They do require the stated intercept-model setup for the familiar sums-of-squares decomposition and interpretations below; assessing regression assumptions for inference is a separate task.

Interpreting R-Squared

For a least-squares model with an intercept, \(R^2\) is the fraction of the observed response variation around \(\bar{y}\) that is accounted for by differences in the fitted values. For example, an \(R^2\) of \(0.70\) means that 70% of the total variation in the response values in this data set is represented by variation in the model’s fitted values. The remaining 30% is the fraction represented by residual variation.

This is an in-sample description, not a statement that the model explains 70% of every individual response, or that 70% of the response is caused by the predictors. A high value can occur even if the model is poorly specified or performs badly for new observations; a low value does not by itself establish that a fitted relationship is unimportant.

Why Adjusted R-Squared Has a Penalty

Ordinary \(R^2\) cannot decrease when a predictor is added to a model fit by least squares to the same observations, provided both models include an intercept. The expanded model can always reproduce the old fitted values by assigning a coefficient of zero to the added predictor. Since least squares chooses coefficients to minimize \(SSE\), the expanded model’s \(SSE\) cannot be larger. The same data have the same \(SST\), so a smaller or unchanged \(SSE\) implies a larger or unchanged \(R^2=1-SSE/SST\).

The adjusted measure accounts for model size by comparing two variance estimates: the residual mean square \(SSE/(n-k-1)\), and the sample variance of the response \(SST/(n-1)\). It subtracts their ratio from 1. Adding slope terms uses up residual degrees of freedom, so a modest improvement in \(SSE\) need not compensate for the penalty. As a result, adjusted R-squared can fall when ordinary R-squared rises.

A Short Justification of the Sums-of-Squares Decomposition

The familiar interpretation of \(R^2\) as the proportion represented by fitted-value variation follows from the intercept and least-squares conditions. Write each observation’s deviation from the response mean as the sum of a fitted-value deviation and a residual:

Squaring and summing over observations gives a cross-product term. In least squares with an intercept, the residuals sum to zero and are orthogonal to every fitted value. Therefore they are also orthogonal to the fitted values after subtracting their mean, and the cross-product is zero:

Consequently, the squared deviations split into fitted-value variation plus residual variation:

Dividing by \(SST>0\) shows that \(SSR/SST=1-SSE/SST=R^2\). This identity depends on the intercept-model setup; without an intercept, the usual centered \(SST\) decomposition and interpretation need not hold.

Using the Measures Carefully

R-squared and adjusted R-squared are useful summaries, but they answer limited questions. R-squared describes the observed fit; adjusted R-squared modifies that summary to account for how many slope terms were estimated relative to the sample size. Neither is a universal measure of model quality. In particular, a higher adjusted value is not proof that the model’s assumptions are correct, that its coefficients are meaningful, or that it will predict new responses well.

Comparisons are most interpretable when models use the same response and the same observations. If the observations differ, their \(SST\) values and response variation may differ as well, making the two summaries less directly comparable. Predictor choice, study design, residual patterns, and the intended use of the model matter in addition to these fit measures.

Common Mistakes

• Calling R-squared a causal percentage. The correction is to say that it summarizes variation in the observed responses represented by fitted values; it does not assign causal responsibility.
• Assuming adjusted R-squared must rise when a predictor is added. Ordinary \(R^2\) cannot fall for nested least-squares models on the same observations, but adjusted \(R^2\) can fall because its degrees-of-freedom penalty changes.
• Treating adjusted R-squared as an ordinary percentage explained. It is a penalized fit summary and can even be negative, so it is not always a literal proportion of observed variation.
• Ignoring the denominator conditions. Adjusted R-squared is undefined when \(n-k-1=0\); verify positive residual degrees of freedom before applying its formula.
• Using one measure to declare a model best. Fit summaries alone do not establish sound assumptions, meaningful effects, or good predictive performance on new data.