Deriving the Sum-of-Squares Decomposition

Why Decompose Variation?

In regression, the observed responses vary around their sample mean. A fitted model assigns some of that variation to differences among its fitted values; the remaining differences are residuals. The sum-of-squares decomposition gives an exact accounting of these two parts when the model is fitted by least squares with an intercept.

The result is an algebraic identity, not a test of whether a model is useful and not a statement that predictors cause changes in the response. Its key ingredient is residual orthogonality: in least squares with an intercept, the residuals have zero sum and are orthogonal to the fitted values. We will derive those properties and then use them to prove the identity.

The Three Sums of Squares

Suppose $n$ observations have responses $y_1,\ldots,y_n$, sample mean $\bar{y}$, fitted values $\hat{y}_1,\ldots,\hat{y}_n$, and residuals $e_i=y_i-\hat{y}_i$. Each of the following sums measures variation in squared response units.

Definition: The total sum of squares, regression sum of squares, and residual sum of squares are, respectively, $$ SS_T=\sum_{i=1}^{n}(y_i-\bar{y})^2,\qquad SS_R=\sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2,\qquad SS_E=\sum_{i=1}^{n}e_i^2. $$ Here $SS_T$ measures total observed variation around the response mean, $SS_R$ measures variation of the fitted values around that mean, and $SS_E$ measures the squared discrepancies between observed and fitted responses.

These definitions distinguish the three quantities by what is being subtracted: the observed responses are centered at $\bar{y}$ for $SS_T$, fitted values are centered at $\bar{y}$ for $SS_R$, and the residuals are squared for $SS_E$. For a model with an intercept, the mean of the fitted values equals $\bar{y}$, but the definitions above make the reference point explicit.

Least-Squares Orthogonality

Consider a linear regression fitted by minimizing the sum of squared residuals over a set of coefficients that includes an intercept. In matrix notation, let $\mathbf{y}$ be the vector of observed responses and let $\mathbf{X}$ be the design matrix, whose columns contain the intercept and the predictors. The fitted vector is $\hat{\mathbf{y}}=\mathbf{X}\hat{\boldsymbol{\beta}}$, and the residual vector is $\mathbf{e}=\mathbf{y}-\hat{\mathbf{y}}$.

No normality or independence assumption is needed for the following algebra. It relies on the least-squares fit and on the design matrix containing a column of ones for the intercept. A least-squares solution minimizes $(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^\mathsf{T}(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})$. At a minimum, the normal equations give $\mathbf{X}^\mathsf{T}\mathbf{e}=\mathbf{0}$: the residual vector is orthogonal to every column of the design matrix.

Key result: For least squares with an intercept, the residuals sum to zero, $\sum_i e_i=0$, and the residual vector is orthogonal to the fitted vector, $\sum_i e_i\hat{y}_i=0$. The first equality follows from orthogonality to the column of ones; the second follows because the fitted vector is a linear combination of columns of the design matrix.

The second property can also be seen directly: $\mathbf{e}^\mathsf{T}\hat{\mathbf{y}}=\mathbf{e}^\mathsf{T}\mathbf{X}\hat{\boldsymbol{\beta}}=(\mathbf{X}^\mathsf{T}\mathbf{e})^\mathsf{T}\hat{\boldsymbol{\beta}}=0$. These two zero sums are exactly what makes the cross-product disappear in the decomposition.

Deriving the Decomposition

For every observation, write its deviation from the response mean as the sum of a fitted-value deviation and a residual:

y_i-\bar{y}=(\hat{y}_i-\bar{y})+e_i.

Squaring and summing this equality creates three terms: the squared fitted-value deviations, the squared residuals, and a cross-product. The cross-product is zero because of residual orthogonality and the zero residual sum.

Expand the square.
For each observation, $(y_i-\bar{y})^2=(\hat{y}_i-\bar{y})^2+e_i^2+2e_i(\hat{y}_i-\bar{y})$.

Sum over observations.
The cross-product contribution is $2\sum_i e_i(\hat{y}_i-\bar{y})=2[\sum_i e_i\hat{y}_i-\bar{y}\sum_i e_i]$.

Apply least-squares orthogonality.
Both $\sum_i e_i\hat{y}_i$ and $\sum_i e_i$ are zero, so the cross-product contribution is zero.

Theorem: For a least-squares regression fit with an intercept, $$ SS_T=SS_R+SS_E, \qquad\text{or equivalently}\qquad \sum_{i=1}^{n}(y_i-\bar{y})^2 = \sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2 + \sum_{i=1}^{n}e_i^2. $$

This proves the identity: summing the expanded squares gives $SS_T=SS_R+SS_E+2\sum_i e_i(\hat{y}_i-\bar{y})$, and the final term is zero by the two orthogonality properties. The identity is exact for the fitted values and residuals from the least-squares calculation. Small discrepancies in hand calculations may arise from rounding intermediate values.

Worked Examples

Worked Example: Checking the Identity in Simple Regression

Suppose a hypothetical dataset contains four observations, with predictor values $x=(0,1,2,3)$ and responses $y=(2,1,4,5)$. Fit a simple least-squares regression with an intercept, then verify the decomposition.

The means are $\bar{x}=1.5$ and $\bar{y}=3$. The slope is $$ \hat{\beta}_1= \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2} = \frac{1.5+1+0.5+3}{2.25+0.25+0.25+2.25} = \frac{6}{5}=1.2. $$ The intercept is $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=3-(1.2)(1.5)=1.2$. Thus $\hat{y}_i=1.2+1.2x_i$, giving fitted values $(1.2,2.4,3.6,4.8)$ and residuals $(0.8,-1.4,0.4,0.2)$.

The total sum of squares is $$ SS_T=(2-3)^2+(1-3)^2+(4-3)^2+(5-3)^2 =1+4+1+4=10. $$ The regression sum of squares is $$ SS_R=(1.2-3)^2+(2.4-3)^2+(3.6-3)^2+(4.8-3)^2 =3.24+0.36+0.36+3.24=7.2. $$ The residual sum of squares is $$ SS_E=0.8^2+(-1.4)^2+0.4^2+0.2^2 =0.64+1.96+0.16+0.04=2.8. $$ Therefore $SS_R+SS_E=7.2+2.8=10=SS_T$. As a check on the cross-product, the fitted-value deviations are $(-1.8,-0.6,0.6,1.8)$, and their products with the residuals sum to $-1.44+0.84+0.24+0.36=0$.

Worked Example: A Multiple Regression Fit

Consider a hypothetical four-observation dataset with two binary predictors. Let $x_1=(0,1,0,1)$, $x_2=(0,0,1,1)$, and $y=(11,11,12,16)$. A possible fitted vector is $(10,12,13,15)$, corresponding to fitted values $10+2x_1+3x_2$. The associated residuals are $(1,-1,-1,1)$.

To confirm that these are least-squares fitted values, check that the residuals are orthogonal to the intercept and both predictor columns: $$ \sum_i e_i=1-1-1+1=0,\quad \sum_i e_ix_{1i}=(-1)+1=0,\quad \sum_i e_ix_{2i}=(-1)+1=0. $$ Thus the residuals are orthogonal to every column in the design matrix, as required by the least-squares normal equations. The fitted values are a combination of those columns, so they are also orthogonal to the residuals.

The response mean is $\bar{y}=(11+11+12+16)/4=12.5$. Hence $$ SS_T=(-1.5)^2+(-1.5)^2+(-0.5)^2+(3.5)^2 =2.25+2.25+0.25+12.25=17. $$ For the fitted values, $$ SS_R=(-2.5)^2+(-0.5)^2+(0.5)^2+(2.5)^2 =6.25+0.25+0.25+6.25=13. $$ For the residuals, $SS_E=1^2+(-1)^2+(-1)^2+1^2=4$. Therefore $SS_R+SS_E=13+4=17=SS_T$. The same accounting works with multiple predictors because orthogonality applies to the entire fitted vector, not only to a single predictor.

Worked Example: The Intercept-Only Boundary Case

Suppose a hypothetical set of responses is $y=(3,5,7)$, and the only fitted term is an intercept. Least squares sets every fitted value to the mean, $\bar{y}=5$, so $\hat{y}=(5,5,5)$ and $e=(-2,0,2)$.

Here $SS_T=(3-5)^2+(5-5)^2+(7-5)^2=4+0+4=8$. Since every fitted value equals the mean, $SS_R=0$. The residual sum of squares is $SS_E=(-2)^2+0^2+2^2=8$. Thus $SS_T=0+8$. This case shows that the decomposition does not require the predictors to explain any variation: it still holds when the regression sum of squares is zero.

Conditions, Interpretation, and a Common Pitfall

The intercept matters because it ensures that the residuals sum to zero and that the fitted values are centered at the response mean. Without an intercept, the same mean-centered identity is not guaranteed. For example, for $x=(1,2)$ and $y=(1,1)$, a least-squares fit constrained through the origin has slope $(1+2)/(1+4)=0.6$, fitted values $(0.6,1.2)$, and residuals $(0.4,-0.2)$. The total sum of squares around $\bar{y}=1$ is zero, while $SS_R=0.2$ and $SS_E=0.2$; their sum is $0.4$, not zero. The intercept-based decomposition therefore cannot be assumed for a fit without an intercept.

Wrong reasoning: “Observed response equals fitted value plus residual, so the squared totals must always add.” Squaring also creates a cross-product; it does not disappear automatically.
Correction: The cross-product vanishes here because least squares with an intercept gives both $\sum_i e_i=0$ and $\sum_i e_i\hat{y}_i=0$. Check the fitting conditions before using the identity.
Wrong interpretation: Treating $SS_R$ as proof that the regression has explained a meaningful or causal amount of variation. The identity is an accounting result; it does not by itself establish practical importance, model adequacy, or causation.

All three sums of squares are measured in squared response units. For example, if a response is measured in milligrams, these sums are in squared milligrams. The decomposition says exactly how the total squared deviations divide for this least-squares fit; it does not say that the fitted component is scientifically important or that the model is appropriate for every purpose.

Check Your Understanding

Use the definitions and orthogonality result to answer these questions.

State the definitions of $SS_T$, $SS_R$, and $SS_E$, and identify the reference value used in each.
Why does a least-squares fit with an intercept have residuals that sum to zero?
In the expansion of $(y_i-\bar{y})^2$, which cross-product must vanish to obtain the decomposition?
Does the decomposition require normally distributed errors? Explain what it requires instead.
What is $SS_R$ for an intercept-only fit, and why does the decomposition still hold?
Why should the mean-centered decomposition not be assumed for a least-squares fit constrained through the origin?