Decomposing Total Variation in Simple Regression

Why Separate Fitted and Residual Variation?

A simple regression describes a response \(y\) using a predictor \(x\). After fitting a line, each observed response has a fitted value and a residual. The fitted values describe the part of the observed responses represented by the line; the residuals describe the differences left over. Separating these two sources of variation gives a useful way to assess and audit a least-squares calculation.

The key result is an exact identity, not an approximation: when ordinary least squares fits a line with an intercept, the total squared deviations of the responses from their mean equal the squared deviations of the fitted values from that same mean plus the residual sum of squares. The identity is algebraic. It does not require normally distributed errors or any other probability assumption.

Definition: For observations \((x_i,y_i)\), \(i=1,\ldots,n\), fit the least-squares line \(\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i\), including an intercept, and define the residual \(e_i=y_i-\hat{y}_i\). Write \(\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i\). The total sum of squares, fitted sum of squares, and residual sum of squares are, respectively,

SS_{\text{Total}}=\sum_{i=1}^{n}(y_i-\bar{y})^2,\qquad SS_{\text{Fit}}=\sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2,\qquad SS_{\text{Res}}=\sum_{i=1}^{n}e_i^2.

These definitions use squared deviations, so each sum is nonnegative and has units equal to the square of the response units. For example, if \(y\) is measured in days, the sums of squares are measured in days squared. The fitted sum of squares is centered on \(\bar{y}\), not on zero; this distinction matters in calculations.

The Decomposition for a Line With an Intercept

The least-squares fit chooses its intercept and slope to minimize the residual sum of squares. The resulting normal equations imply two facts: the residuals sum to zero, and the residuals have zero sum of products with the predictor values. Those facts make the fitted values orthogonal to the residuals, in the sense that their sum of products is zero.

Theorem: For an ordinary least-squares line with an intercept,

SS_{\text{Total}}=SS_{\text{Fit}}+SS_{\text{Res}}.

This identity applies to the observed responses and the fitted values from the same least-squares fit. It does not require a probability model for the errors.

Here is a direct justification. Because the least-squares line includes an intercept, its normal equations give \(\sum e_i=0\) and \(\sum x_i e_i=0\). Since each fitted value is \(\hat{\beta}_0+\hat{\beta}_1x_i\), it follows that

\sum_{i=1}^{n}\hat{y}_i e_i =\hat{\beta}_0\sum_{i=1}^{n}e_i+\hat{\beta}_1\sum_{i=1}^{n}x_i e_i =0.

Also, \(\bar{\hat{y}}=\bar{y}\): the average fitted value equals the average response. This follows from \(\sum e_i=0\) and \(y_i=\hat{y}_i+e_i\). Therefore, for each observation,

y_i-\bar{y}=(\hat{y}_i-\bar{y})+e_i.

Square this equation and sum over observations. The result is

\begin{aligned} \sum_{i=1}^{n}(y_i-\bar{y})^2 &=\sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2 +\sum_{i=1}^{n}e_i^2 +2\sum_{i=1}^{n}(\hat{y}_i-\bar{y})e_i. \end{aligned}

The cross-product sum is zero, since

\sum_{i=1}^{n}(\hat{y}_i-\bar{y})e_i =\sum_{i=1}^{n}\hat{y}_i e_i-\bar{y}\sum_{i=1}^{n}e_i =0-0=0.

The remaining terms are exactly \(SS_{\text{Fit}}\) and \(SS_{\text{Res}}\), proving the identity. This derivation also explains why adding the two sums works: the fitted deviations and residuals contribute no cross-product overall.

Worked Example: Decomposing Variation in a Four-Observation Fit

A hypothetical clinic records patient load \(x\), in tens of appointments per day, and average waiting time \(y\), in minutes, on four days. The observations are \((0,1),(1,3),(2,4),(3,6)\). Fit the least-squares line with an intercept, then calculate and check the three sums of squares. The values are illustrative, not industry data.

Method and fit. First calculate the means: \(\bar{x}=(0+1+2+3)/4=1.5\) and \(\bar{y}=(1+3+4+6)/4=3.5\). The centered predictor sum of squares is

S_{xx}=(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2 =2.25+0.25+0.25+2.25=5.

The centered cross-product sum is \(S_{xy}=(-1.5)(-2.5)+(-0.5)(-0.5)+(0.5)(0.5)+(1.5)(2.5)=3.75+0.25+0.25+3.75=8\). Thus \(\hat{\beta}_1=S_{xy}/S_{xx}=8/5=1.6\), and \(\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=3.5-(1.6)(1.5)=1.1\). The fitted line is \(\hat{y}=1.1+1.6x\).

Fitted values and residuals. At \(x=0,1,2,3\), the fitted values are \(1.1,2.7,4.3,5.9\). Subtract each fitted value from its observed response to obtain residuals \(-0.1,0.3,-0.3,0.1\). Their sum is \(-0.1+0.3-0.3+0.1=0\), as expected for a fit with an intercept.

Total variation. The response deviations from \(3.5\) are \(-2.5,-0.5,0.5,2.5\). Therefore

SS_{\text{Total}}=(-2.5)^2+(-0.5)^2+(0.5)^2+(2.5)^2 =6.25+0.25+0.25+6.25=13.

Fitted and residual variation. The fitted-value deviations from \(3.5\) are \(-2.4,-0.8,0.8,2.4\), so \(SS_{\text{Fit}}=5.76+0.64+0.64+5.76=12.8\). The squared residuals sum to \(SS_{\text{Res}}=0.01+0.09+0.09+0.01=0.20\).

Check. The two components add to \(12.8+0.20=13=SS_{\text{Total}}\). The decomposition holds. The common mistake this calculation avoids is centering the response for the total sum but forgetting to center the fitted values for the fitted sum.

What the Components Measure

The total sum of squares measures how dispersed the observed responses are around their sample mean, without using the predictor. The fitted sum of squares measures how dispersed the fitted values are around that same mean. The residual sum of squares measures the squared distances from the observed responses to their fitted values. The identity says that, for this least-squares fit with an intercept, the total centered variation is accounted for by these two components.

This separation is meaningful because the fitted line has been chosen by least squares. It is not a general identity for any candidate line. For a line that is not the least-squares fit, the residuals need not sum to zero or be orthogonal to the fitted values, and a cross-product term may remain when the squared deviations are expanded.

The intercept is also essential to this centered version of the result. It ensures that the fitted values have mean \(\bar{y}\), and that the residuals sum to zero. If a regression is fit without an intercept, these centered properties need not hold, so the same decomposition of deviations from \(\bar{y}\) is not guaranteed. The identity here should be used only when the model is fit by least squares with an intercept.

Worked Example: Checking the Decomposition and Avoiding a Centering Error

A hypothetical service team observes workload \(x=1,2,3,4\) and response time \(y=2,2,5,7\), in consistent units. Find the fitted line and all three sums of squares. Then examine a tempting but incorrect calculation using uncentered fitted values.

Method and fitted line. The means are \(\bar{x}=2.5\) and \(\bar{y}=4\). The predictor sum of squares is \(S_{xx}=(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2=5\). The cross-product sum is \(S_{xy}=(-1.5)(-2)+(-0.5)(-2)+(0.5)(1)+(1.5)(3)=3+1+0.5+4.5=9\). Hence \(\hat{\beta}_1=9/5=1.8\), and \(\hat{\beta}_0=4-(1.8)(2.5)=-0.5\). The fitted line is \(\hat{y}=-0.5+1.8x\).

Fitted values and residuals. The fitted values are \(1.3,3.1,4.9,6.7\), and the residuals are \(0.7,-1.1,0.1,0.3\). Their squared sum is \(0.49+1.21+0.01+0.09=1.8\), so \(SS_{\text{Res}}=1.8\).

The observed responses have deviations \(-2,-2,1,3\) from \(\bar{y}=4\), giving \(SS_{\text{Total}}=4+4+1+9=18\). The fitted values have deviations \(-2.7,-0.9,0.9,2.7\), giving \(SS_{\text{Fit}}=7.29+0.81+0.81+7.29=16.2\). Thus \(16.2+1.8=18\), which checks the decomposition.

Common mistake and correction. A wrong calculation uses the uncentered fitted values and claims that \(SS_{\text{Fit}}=\sum \hat{y}_i^2=1.69+9.61+24.01+44.89=80.2\). This is not the defined fitted sum of squares: it measures squared distance from zero, not from \(\bar{y}=4\). The correct fitted sum is \(80.2-4(4^2)=80.2-64=16.2\), equivalently the sum of squared deviations from \(4\). Similarly, \(\sum y_i^2=82\) is not \(SS_{\text{Total}}\); centering gives \(82-4(4^2)=18\). Do not combine a raw sum of squares with centered components.

Worked Example: Verifying the Components From Orthogonal Residuals

For a hypothetical equipment-monitoring study, let the predictor values be \(x=0,1,2,3\) and the observed responses be \(y=2,2,4,8\). Find the least-squares fit, show that the residuals are orthogonal to the fitted values, and verify the variation identity.

Fit the line. Here \(\bar{x}=1.5\) and \(\bar{y}=4\). As in the direct calculation, \(S_{xx}=5\). The cross-product sum is \((-1.5)(-2)+(-0.5)(-2)+(0.5)(0)+(1.5)(4)=3+1+0+6=10\). Therefore \(\hat{\beta}_1=10/5=2\) and \(\hat{\beta}_0=4-(2)(1.5)=1\). The fitted values are \(1,3,5,7\).

Residual checks. Subtracting the fitted values from the observations gives residuals \(1,-1,-1,1\). Their sum is \(1-1-1+1=0\). Their predictor-weighted sum is \((0)(1)+(1)(-1)+(2)(-1)+(3)(1)=-1-2+3=0\). Thus the residuals are orthogonal to the fitted values: \(1(1)+3(-1)+5(-1)+7(1)=1-3-5+7=0\).

Calculate and check the sums. The observed deviations from \(4\) are \(-2,-2,0,4\), so \(SS_{\text{Total}}=4+4+0+16=24\). The fitted deviations from \(4\) are \(-3,-1,1,3\), so \(SS_{\text{Fit}}=9+1+1+9=20\). The squared residuals sum to \(SS_{\text{Res}}=1+1+1+1=4\). Finally, \(20+4=24\), as required.

The method illustrates the reason the identity works: the fitted deviations and residuals have zero sum of products, so expanding the total squared deviations creates no extra cross-product contribution.

Common Interpretation and Calculation Traps

Using the wrong center. Both \(SS_{\text{Total}}\) and \(SS_{\text{Fit}}\) use deviations from \(\bar{y}\). The residual sum uses deviations from the fitted line, \(e_i=y_i-\hat{y}_i\).
Using a non-least-squares line. The decomposition depends on the least-squares normal equations. A line chosen by another method may leave a nonzero cross-product term.
Omitting the intercept. The centered identity is not guaranteed for a fit without an intercept. Check which model was actually fit before applying the result.
Confusing a numerical check with a definition. A tiny discrepancy can result from rounding fitted values too early. Retain full precision through the sums, then round only the final reported values.
Assuming a distributional condition is needed. The decomposition is an algebraic property of the least-squares fit with an intercept. Distributional assumptions matter for inference, not for this identity.

This decomposition is useful for checking regression arithmetic and for distinguishing variation represented by the fitted line from variation remaining in the residuals. In later calculations, the same sums can be combined into further summaries, but their definitions must remain consistent: total and fitted variation are centered on the response mean, while residual variation is measured from the fitted values.

Key takeaway: For an ordinary least-squares regression line with an intercept, \(SS_{\text{Total}}=SS_{\text{Fit}}+SS_{\text{Res}}\). The equality follows because residuals sum to zero and are orthogonal to fitted values. Keep all sums centered as defined, and do not assume the centered identity applies to a fit without an intercept.

Check Your Understanding

For each problem, use the definitions in this tutorial and check that the fitted and residual components add to the total.

A least-squares line with an intercept is fit to \(x=0,1,2\) and \(y=1,2,4\). Find the fitted line, \(SS_{\text{Total}}\), \(SS_{\text{Fit}}\), and \(SS_{\text{Res}}\). (Answer: \(\hat{y}=5/6+(3/2)x\); \(SS_{\text{Total}}=14/3\), \(SS_{\text{Fit}}=9/2\), \(SS_{\text{Res}}=1/6\).)
For a fit with an intercept, the observed responses are \(3,3,4,6\) at predictor values \(1,2,3,4\). Calculate the fitted line and the three sums of squares. (Answer: \(\hat{y}=1.5+x\); \(SS_{\text{Total}}=6\), \(SS_{\text{Fit}}=5\), \(SS_{\text{Res}}=1\).)
A fitted line with an intercept has \(n=6\), \(\bar{y}=12\), \(\sum y_i^2=914\), \(\sum \hat{y}_i^2=900\), and \(SS_{\text{Res}}=14\). Find \(SS_{\text{Total}}\) and \(SS_{\text{Fit}}\), then check the identity. (Answer: \(SS_{\text{Total}}=50\), \(SS_{\text{Fit}}=36\), and \(36+14=50\).)
For a least-squares fit with an intercept, \(S_{xx}=9\), \(S_{xy}=18\), and \(SS_{\text{Res}}=14\). Find the fitted slope and the fitted and total sums of squares. (Answer: \(\hat{\beta}_1=2\), \(SS_{\text{Fit}}=S_{xy}^2/S_{xx}=36\), and \(SS_{\text{Total}}=50\).)
An analyst reports \(SS_{\text{Total}}=40\) and \(SS_{\text{Res}}=11\) for an ordinary least-squares line with an intercept, but reports a fitted sum of squares of \(31\). Determine whether the reported values are consistent. State the missing fitted sum required by the identity. (Answer: They are inconsistent; \(SS_{\text{Fit}}\) must be \(40-11=29\).)

What You'll Learn