R-Squared, Residual Variation, and Model Expansion

Why Separate Fit From Explanation?

A regression can account for a large share of the observed variation in an outcome and still be a poor guide to what would happen under an intervention or in new data. To understand what a fit statistic does—and does not—say, begin with the exact relationship between fitted values and residuals in ordinary least squares (OLS) with an intercept.

This relationship also explains a simple fact about model expansion: when a regression includes all the regressors in another regression, plus additional regressors, its in-sample \(R^2\) cannot be smaller. That mathematical guarantee is about fitting the observations used to estimate the model. It is not a guarantee of better prediction elsewhere or of a valid causal interpretation.

OLS With an Intercept and the Centered Decomposition

For observations \(i=1,\ldots,n\), let \(y_i\) be the outcome and let \(x_i\) denote a vector of regressors. OLS with an intercept chooses coefficients to minimize the sum of squared residuals. Write the fitted value as \(\hat{y}_i\), the residual as \(\hat{e}_i=y_i-\hat{y}_i\), and the outcome sample mean as \(\bar{y}=n^{-1}\sum_{i=1}^n y_i\). The intercept is included among the regressors.

Define the total sum of squares (SST), regression sum of squares (SSR), and error sum of squares (SSE) by

\mathrm{SST}=\sum_{i=1}^n(y_i-\bar{y})^2,\qquad \mathrm{SSR}=\sum_{i=1}^n(\hat{y}_i-\bar{y})^2,\qquad \mathrm{SSE}=\sum_{i=1}^n\hat{e}_i^2.

SST measures the outcome's total variation around its sample mean. SSE measures the squared residual variation left after fitting the model. With an intercept, SSR measures the fitted values' variation around that same mean.

Theorem: For OLS estimated with an intercept on a sample with \(\mathrm{SST}>0\), the centered sums of squares satisfy \(\mathrm{SST}=\mathrm{SSR}+\mathrm{SSE}\). The usual coefficient of determination is \(R^2=\mathrm{SSR}/\mathrm{SST}=1-\mathrm{SSE}/\mathrm{SST}\), and therefore lies between 0 and 1.

The key OLS properties behind the identity are that the residuals sum to zero and are orthogonal to the fitted values: \(\sum_i\hat{e}_i=0\) and \(\sum_i\hat{e}_i\hat{y}_i=0\). These follow from the least-squares normal equations when the model includes an intercept; the second also follows because fitted values lie in the model's regressor space and the residual vector is orthogonal to that space.

To derive the decomposition, write each deviation from the outcome mean as \(y_i-\bar{y}=(\hat{y}_i-\bar{y})+\hat{e}_i\). Squaring and summing gives

\begin{aligned} \sum_{i=1}^n(y_i-\bar{y})^2 &=\sum_{i=1}^n(\hat{y}_i-\bar{y})^2 +\sum_{i=1}^n\hat{e}_i^2 +2\sum_{i=1}^n(\hat{y}_i-\bar{y})\hat{e}_i\\ &=\mathrm{SSR}+\mathrm{SSE} +2\left(\sum_{i=1}^n\hat{y}_i\hat{e}_i-\bar{y}\sum_{i=1}^n\hat{e}_i\right)\\ &=\mathrm{SSR}+\mathrm{SSE}. \end{aligned}

Both terms in parentheses are zero by the OLS properties above. Dividing the decomposition by SST gives the two equivalent expressions for \(R^2\). Because SSR and SSE are sums of squares, neither is negative; when SST is positive, \(0\leq R^2\leq1\).

Worked Example: Decomposing Variation in a Small Regression

Suppose, hypothetically, four students' quiz scores \(y_i\), in points, are recorded alongside their study hours \(x_i\). The observations are \((0,1),(1,2),(2,2),(3,5)\). Fit an OLS line with an intercept and calculate its centered sums of squares and \(R^2\).

The sample means are \(\bar{x}=1.5\) hours and \(\bar{y}=2.5\) points. The OLS slope and intercept are

\hat{\beta}_1= \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2} =\frac{6}{5}=1.2,\qquad \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=2.5-(1.2)(1.5)=0.7.

Thus the fitted line is \(\hat{y}_i=0.7+1.2x_i\). In the order of the observations, the fitted values are \(0.7,1.9,3.1,4.3\), and the residuals are \(0.3,0.1,-1.1,0.7\). Their sum is zero, as required by the intercept normal equation.

The total and residual sums of squares are

\begin{aligned} \mathrm{SST} &=(-1.5)^2+(-0.5)^2+(-0.5)^2+(2.5)^2=9,\\ \mathrm{SSE} &=(0.3)^2+(0.1)^2+(-1.1)^2+(0.7)^2=1.8. \end{aligned}

The fitted-value deviations from \(\bar{y}\) are \(-1.8,-0.6,0.6,1.8\), so \(\mathrm{SSR}=(-1.8)^2+(-0.6)^2+(0.6)^2+(1.8)^2=7.2\). The decomposition checks: \(9=7.2+1.8\). Hence \(R^2=7.2/9=1-1.8/9=0.800\). In this sample, 80% of the quiz-score variation around the sample mean is accounted for by the fitted line.

Why Adding Regressors Cannot Lower In-Sample R-Squared

Consider two OLS models fitted to the same outcome observations. The expanded model contains every regressor in the original model and at least one additional regressor; both models include an intercept. Their regressor spaces are nested: every fitted-value vector achievable in the original model is also achievable in the expanded model, by setting the added coefficients to zero.

OLS selects, from its model's available fitted-value vectors, the one that minimizes SSE. Since the expanded model has all the original choices and possibly more, its minimum SSE cannot be larger. SST is unchanged because the outcome observations are unchanged. Using \(R^2=1-\mathrm{SSE}/\mathrm{SST}\), a weakly smaller SSE therefore means a weakly larger \(R^2\). An added regressor can leave \(R^2\) unchanged, for example when it contributes no new direction to the fitted-value space.

Result: For nested OLS models with an intercept, estimated on the same observations with the same outcome, adding regressors cannot reduce in-sample \(R^2\). This result does not require the added regressors to be statistically significant or substantively useful.

Worked Example: Calculating the Increase After Adding a Regressor

Use the four hypothetical observations from the first example. The original model has an intercept and study hours \(x=(0,1,2,3)\), with \(\mathrm{SSE}=1.8\) and \(\mathrm{SST}=9\). Now add \(z=(0,0,1,0)\), an indicator equal to one for the third observation and zero otherwise. Calculate the new SSE and \(R^2\).

First remove from \(z\) the part explained by the original model. Regressing \(z\) on an intercept and \(x\) gives fitted values \(0.1,0.2,0.3,0.4\), so the residualized regressor is \(q=(-0.1,-0.2,0.7,-0.4)\). The original-model residual vector is \(e=(0.3,0.1,-1.1,0.7)\). The additional regressor's component orthogonal to the original model can reduce SSE by

\frac{(q'e)^2}{q'q} =\frac{(-1.10)^2}{0.70} =\frac{1.21}{0.70} =1.7286.

Here \(q'e=(-0.1)(0.3)+(-0.2)(0.1)+(0.7)(-1.1)+(-0.4)(0.7)=-1.10\), and \(q'q=0.01+0.04+0.49+0.16=0.70\). Thus the expanded model has \(\mathrm{SSE}=1.8-1.7286=0.07143\) (rounding). Its \(R^2\) is \(1-0.07143/9=0.9921\), greater than the original \(0.800\). The increase describes fit to these four observations; by itself, it does not show that the added indicator will help on new observations.

What R-Squared Does Not Establish

A high \(R^2\) is a statement about in-sample variation around the outcome mean, not a general certificate of model quality. In particular, it does not establish that a regression coefficient is causal. A fitted association can arise because the regressor is related to other determinants of the outcome, or because the model has other shortcomings. Causal interpretation requires an appropriate identification argument and assumptions beyond a good fit statistic.

Nor does high in-sample \(R^2\) guarantee useful prediction. A model can fit idiosyncrasies in its estimation sample that do not recur in new data. Predictive performance concerns prediction errors for observations not used to fit the model, so it requires an appropriate assessment on new or held-out data. A rise in training-sample \(R^2\) alone does not provide that assessment.

Worked Example: Perfect Fit Does Not Identify a Causal Effect

Suppose a hypothetical observational data set records a treatment \(x\), measured in hours, and an outcome \(y\), measured in score points: \((1,12),(2,14),(3,16),(4,18)\). The OLS line with an intercept is \(\hat{y}=10+2x\). Since every observation lies exactly on the line, \(\mathrm{SSE}=0\), \(\bar{y}=15\), and \(\mathrm{SST}=(-3)^2+(-1)^2+1^2+3^2=20\). Therefore \(R^2=1-0/20=1\).

The fitted slope describes a two-point increase in the observed outcome for each additional observed treatment hour in this sample. But the data alone do not establish that changing treatment hours would cause that increase. For example, the same observed pattern is compatible with an unmeasured factor \(U\) taking values \(1,2,3,4\), with \(x=U\) and \(y=10+2U\), where the outcome is determined by \(U\) rather than by an effect of \(x\). This alternative produces the same observed pairs, yet does not imply that intervening to change \(x\) changes \(y\). The perfect \(R^2\) cannot distinguish these explanations.

Perfect fit in these four observations also says nothing by itself about prediction for a new observation. The exact linear pattern may not continue beyond the observed sample. Thus the computed \(R^2\) is accurate, but its scope is limited to fit in these data.

Common Mistakes and Practical Interpretation

Claiming added regressors must improve the model. The guarantee is only that in-sample SSE cannot rise and \(R^2\) cannot fall in a nested comparison on the same observations. It does not guarantee better out-of-sample prediction or a more credible explanation.
Calling \(R^2\) the percentage of outcomes correctly predicted. \(R^2\) is the fraction of centered outcome variation accounted for by the fitted values under this decomposition. It is not an accuracy rate, and it does not give the size of typical prediction errors in outcome units.
Interpreting high \(R^2\) as causal evidence. Fit describes observed covariation. A causal interpretation requires assumptions and a research design that address why the regressor varies and what else may determine the outcome.
Using the centered decomposition without an intercept. The stated identity uses OLS with an intercept. In a regression without an intercept, residuals need not sum to zero and the usual centered decomposition and \(R^2\) interpretation need not hold.
Comparing \(R^2\) across different samples as if only the model changed. The nondecrease result requires the same observations and outcome. When the sample changes, SST also changes, so that guarantee does not apply.

Key takeaway: With an intercept, OLS partitions centered outcome variation into fitted and residual sums of squares. Adding regressors to a nested model on the same sample cannot reduce in-sample \(R^2\), but that mechanical improvement is not proof of causal validity or useful prediction.

Check Your Understanding

Answer each question using the definitions and conditions given above.

In an OLS model with an intercept, \(\mathrm{SST}=40\) and \(\mathrm{SSE}=10\). Find SSR and \(R^2\), and interpret \(R^2\) in words.
Why does adding a regressor to a nested model, estimated on the same observations, weakly increase \(R^2\)? State the minimizing quantity and the denominator that stays fixed.
Can \(R^2\) stay unchanged after a regressor is added? Give a reason.
A researcher reports that adding controls raised \(R^2\). What does this establish, and what does it not establish about causal validity?
Why does the usual centered sum-of-squares decomposition require care when a regression has no intercept?