Decomposing Variation in Simple Linear Regression

Why Divide Variation Into Parts?

A response variable varies across observations. In simple linear regression, some of that variation is reflected in how the fitted responses differ from their overall mean; the rest appears as differences between observed and fitted responses. Separating these quantities gives a precise accounting of the fit on the data used to estimate the line.

This accounting is an exact identity for ordinary least-squares regression with an intercept. It is not a claim that the model explains every pattern in the data, nor does it by itself show that a line is appropriate. It is an algebraic result that depends on how the line is fitted.

Definition: For \(n\) paired observations \((x_i,y_i)\), let \(\bar{y}\) be the sample mean of the responses, \(\hat{y}_i\) the fitted response for observation \(i\), and \(e_i=y_i-\hat{y}_i\) its residual. Define the total sum of squares, fitted sum of squares, and residual sum of squares by

SST=\sum_{i=1}^{n}(y_i-\bar{y})^2,\qquad SS_{\mathrm{fit}}=\sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2,\qquad SSE=\sum_{i=1}^{n}e_i^2.

SST measures total response variation around the response mean, \(SS_{\mathrm{fit}}\) measures variation among fitted responses around that mean, and SSE measures squared residual variation. Each sum is measured in squared response units.

The Decomposition for a Least-Squares Line

The simple linear regression line has the form \(\hat{y}_i=b_0+b_1x_i\), where \(b_0\) is the intercept and \(b_1\) is the slope. The ordinary least-squares line with an intercept minimizes \(\sum_{i=1}^{n}(y_i-b_0-b_1x_i)^2\) over the intercept and slope. Two properties of this fit are central: its residuals sum to zero, and its residuals have zero sum of products with the fitted responses centered at their mean.

Theorem: For ordinary least-squares regression of \(y\) on \(x\) with an intercept, using the same \(n\) observations to calculate all three sums of squares,

SST=SS_{\mathrm{fit}}+SSE.

The fitted responses have mean \(\bar{y}\), and the cross-product between centered fitted responses and residuals is zero. These properties make the decomposition hold exactly, apart from rounding in reported calculations.

Here is the algebra behind the result. For each observation, add and subtract its fitted response:

y_i-\bar{y}=(\hat{y}_i-\bar{y})+e_i.

Squaring and summing produces the two sums of squares and a cross-product. The cross-product vanishes because of the least-squares properties:

\begin{aligned} SST &=\sum_{i=1}^{n}\left[(\hat{y}_i-\bar{y})+e_i\right]^2\\ &=\sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2 +\sum_{i=1}^{n}e_i^2 +2\sum_{i=1}^{n}(\hat{y}_i-\bar{y})e_i. \end{aligned}

To see why the last sum is zero, first note that an intercept in the least-squares model implies \(\sum e_i=0\). Also, \(\hat{y}_i=b_0+b_1x_i\), so

\begin{aligned} \sum_{i=1}^{n}(\hat{y}_i-\bar{y})e_i &=\sum_{i=1}^{n}\hat{y}_i e_i-\bar{y}\sum_{i=1}^{n}e_i\\ &=\sum_{i=1}^{n}(b_0+b_1x_i)e_i\\ &=b_0\sum_{i=1}^{n}e_i+b_1\sum_{i=1}^{n}x_i e_i\\ &=0. \end{aligned}

The final equality uses both least-squares normal equations: \(\sum e_i=0\) and \(\sum x_i e_i=0\). Substituting the zero cross-product into the expanded expression gives \(SST=SS_{\mathrm{fit}}+SSE\). This derivation uses the fitted line’s least-squares properties and its intercept; it does not require a normality assumption for the responses.

Worked Example: Checking the Decomposition From Data

Suppose four hypothetical observations have predictor values \(x=1,2,3,4\) and response values \(y=2,3,3,6\). Find the least-squares line with an intercept, then calculate each sum of squares and check the identity.

The means are \(\bar{x}=2.5\) and \(\bar{y}=3.5\). The slope is

b_1= \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} =\frac{6}{5}=1.2.

Thus \(b_0=\bar{y}-b_1\bar{x}=3.5-(1.2)(2.5)=0.5\), and the fitted line is \(\hat{y}=0.5+1.2x\). Its fitted responses are \(1.7,2.9,4.1,5.3\), and the residuals \(y_i-\hat{y}_i\) are \(0.3,0.1,-1.1,0.7\). Their sum is zero, as expected for a least-squares fit with an intercept.

The total sum of squares is

$$ SST=(2-3.5)^2+(3-3.5)^2+(3-3.5)^2+(6-3.5)^2 =2.25+0.25+0.25+6.25=9. $$

The fitted sum of squares is

SS_{\mathrm{fit}}=(1.7-3.5)^2+(2.9-3.5)^2+(4.1-3.5)^2+(5.3-3.5)^2 =3.24+0.36+0.36+3.24=7.20.

The residual sum of squares is

$$ SSE=(0.3)^2+(0.1)^2+(-1.1)^2+(0.7)^2 =0.09+0.01+1.21+0.49=1.80. $$

Therefore \(SS_{\mathrm{fit}}+SSE=7.20+1.80=9.00=SST\). The response is measured in its original units, so each of these sums of squares is in squared response units.

What Each Part Counts

The identity says that, for the observations used to fit the line, the squared distance of the responses from their mean divides into two perpendicular components: squared distance of the fitted values from that mean, and squared distance of the observations from their fitted values. The zero cross-product established in the derivation is what allows these two components to add without an extra term.

This is an accounting of variation, not a separation into unrelated causes. The fitted component is variation represented by the line in this particular data set. The residual component is variation left between the observed responses and their fitted values. A small residual sum of squares does not alone establish that a linear model is suitable; residual patterns and the data-collection context still matter.

Worked Example: Reading the Sums in Context

Suppose a hypothetical least-squares model predicts a patient’s follow-up measurement, in millimeters, from a baseline measurement. For a sample of observations, suppose \(SST=144\) square millimeters and \(SS_{\mathrm{fit}}=100\) square millimeters. Find SSE and describe what the three quantities represent.

Because the model is a least-squares regression line with an intercept, the decomposition applies:

SSE=SST-SS_{\mathrm{fit}}=144-100=44\ \text{square millimeters}.

In this sample, the squared deviations of the observed follow-up measurements from their sample mean add to \(144\) square millimeters. The squared deviations of the fitted measurements from that same mean add to \(100\) square millimeters. The squared residuals add to \(44\) square millimeters. These are sums of squared deviations, not amounts of follow-up measurement in millimeters.

The identity allows the missing sum to be calculated, but it does not say that a particular patient’s prediction is accurate or that the predictor causes changes in the response. Those questions require other evidence and checks.

When the Identity Does Not Apply

The exact decomposition depends on fitting an ordinary least-squares line with an intercept to the observations being summarized. If predictions come from a different method, from a line forced through the origin, or from a different set of data, the residuals need not have the required zero-sum and zero cross-product properties. The same algebra still expands, but a nonzero cross-product can remain.

Worked Example: Why a No-Intercept Fit Can Fail the Identity

Consider two hypothetical response values \(y_1=y_2=2\). Their mean is \(\bar{y}=2\), so \(SST=0\). Suppose predictions are \(\hat{y}_1=1.2\) and \(\hat{y}_2=2.4\), as can arise from a least-squares line constrained to pass through the origin for suitable predictor values.

The fitted sum of squares is

SS_{\mathrm{fit}}=(1.2-2)^2+(2.4-2)^2 =(-0.8)^2+(0.4)^2=0.64+0.16=0.80.

The residuals are \(e_1=2-1.2=0.8\) and \(e_2=2-2.4=-0.4\), so

$$ SSE=(0.8)^2+(-0.4)^2=0.64+0.16=0.80. $$

Thus \(SST=0\), \(SS_{\mathrm{fit}}=0.80\), and \(SSE=0.80\); the proposed identity would say \(0=0.80+0.80\), which is false. There is no contradiction: these predictions are not from the ordinary least-squares line with an intercept. In fact, the residuals sum to \(0.8-0.4=0.4\), not zero, so the key condition used in the derivation fails.

Common Mistakes and a Practical Check

Using the wrong center. SST and \(SS_{\mathrm{fit}}\) are both calculated around the response mean \(\bar{y}\), not around zero or the mean predictor value.
Mixing up residuals and deviations from the mean. Residuals are \(e_i=y_i-\hat{y}_i\); total deviations are \(y_i-\bar{y}\). They answer different questions.
Assuming any predictions satisfy the identity. The theorem is for ordinary least-squares regression with an intercept on the same observations. For other predictions, calculate the sums directly rather than forcing the identity.
Reporting squared units as original units. If \(y\) is measured in millimeters, sums of squares are measured in square millimeters.
Interpreting the sum as a causal explanation. The decomposition is a mathematical description of variation in the fitted data, not evidence that changes in \(x\) cause changes in \(y\).

In a calculation, verify that the fitted line includes an intercept and that all sums use the same observations. Then compute the three sums from their definitions. If the equality is slightly off, check arithmetic and rounding; if it is substantially off, check the fit type, the center used, and whether the same cases were included in each sum.

Check Your Understanding

Use the definitions and the conditions for the identity to answer these questions.

In a least-squares regression with an intercept, what does each of SST, \(SS_{\mathrm{fit}}\), and SSE measure?
Why does the cross-product \(\sum(\hat{y}_i-\bar{y})e_i\) equal zero for the fitted line?
A model has \(SST=80\) and \(SSE=23\). If the model is an ordinary least-squares regression with an intercept on the same observations, find \(SS_{\mathrm{fit}}\).
If the response is recorded in kilograms, what are the units of SST and SSE?
Why is the identity not guaranteed for a line forced through the origin?