Why Organize Regression Results in an ANOVA Table?
A regression ANOVA table summarizes how the variation in an observed response is divided between variation accounted for by the fitted regression and variation left in the residuals. Its rows organize these sums of squares alongside degrees of freedom, mean squares, and, for the regression row, the overall F statistic. Constructing the table is useful both for understanding a model's overall test and for checking that reported results agree with one another.
The table below concerns ordinary least squares regression with an intercept. It is important to state that condition: the usual centered total sum of squares and the standard regression and residual degrees of freedom described here rely on fitting an intercept. The table is not a substitute for checking whether the regression assumptions are reasonable.
The Standard Regression ANOVA Table
Suppose there are \(n\) observations and \(p\) predictors. Write the model as \(y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i\). For the usual exact overall F test, assume the errors are independent and normally distributed with mean zero and common variance \(\sigma^2\), and that the predictor design has full column rank. The model includes an intercept, and \(n>p+1\), so there are residual degrees of freedom.
Under these conditions, the standard table has three sources: regression, residual (or error), and total. Degrees of freedom count the independent pieces of information associated with each source. Mean square is the sum of squares divided by its degrees of freedom. The total row ordinarily has no mean square or F statistic.
| Source | Degrees of freedom | Sum of squares | Mean square | F statistic |
|---|---|---|---|---|
| Regression | \(p\) | \(SSR\) | \(MSR=SSR/p\) | \(F=MSR/MSE\) |
| Residual (error) | \(n-p-1\) | \(SSE\) | \(MSE=SSE/(n-p-1)\) | — |
| Total | \(n-1\) | \(SST\) | — | — |
The overall regression F statistic tests \(H_0:\beta_1=\cdots=\beta_p=0\) against the alternative that at least one slope is nonzero. The intercept is not included in this null hypothesis. Under the null and the stated normal-error assumptions, the statistic has an \(F\) distribution with \(p\) numerator and \(n-p-1\) denominator degrees of freedom. This is an overall test: it does not identify which predictor contributes evidence against the null.
Why the Sums of Squares Add
The decomposition follows from the ordinary least squares residuals being orthogonal to the fitted values after their means are removed. In an intercept model, the fitted values average to \(\bar{y}\), and the residuals sum to zero. For each observation, write \(y_i-\bar{y}=(\hat{y}_i-\bar{y})+(y_i-\hat{y}_i)\). Squaring and summing gives the identity, as shown below.
The cross-product sum is zero. Least squares residuals are orthogonal to every fitted linear combination of the model's predictors, including the fitted values; they are also orthogonal to the constant vector because the model includes an intercept. Consequently, their cross-product with \(\hat{y}_i-\bar{y}\) sums to zero. The remaining terms are exactly \(SSR\) and \(SSE\), so \(SST=SSR+SSE\). Without an intercept, these centered definitions and this standard decomposition do not generally apply in the same way; use the definitions and test appropriate to that model rather than forcing this table onto it.
Worked Examples
Worked Example: Constructing a Table from Observations
A hypothetical analyst regresses a response \(y\), measured in index points, on one predictor for five observations. The data are \((x,y)=(1,2),(2,3),(3,5),(4,4),(5,6)\). Construct the full ANOVA table and calculate the overall F statistic.
Method. Calculate the centered sums of squares, then assign degrees of freedom and divide each sum of squares by its degrees of freedom to obtain the mean squares.
First, \(\bar{x}=(1+2+3+4+5)/5=3\), and \(\bar{y}=(2+3+5+4+6)/5=4\). The centered \(x\) values are \(-2,-1,0,1,2\), and the centered \(y\) values are \(-2,-1,1,0,2\). Thus \(S_{xx}=4+1+0+1+4=10\), and \(S_{xy}=4+1+0+0+4=9\). The fitted slope is \(b_1=S_{xy}/S_{xx}=9/10=0.9\).
The fitted values are \(\hat{y}_i=\bar{y}+b_1(x_i-\bar{x})\), giving \(2.2,3.1,4.0,4.9,5.8\). The residuals, observed minus fitted, are \(-0.2,-0.1,1.0,-0.9,0.2\). Therefore \(SSE=0.04+0.01+1.00+0.81+0.04=1.90\). The centered observed responses give \(SST=4+1+1+0+4=10\), so \(SSR=SST-SSE=10-1.90=8.10\). Check: \(8.10+1.90=10.00\).
Here \(n=5\) and \(p=1\). The regression, residual, and total degrees of freedom are \(1\), \(5-1-1=3\), and \(5-1=4\), respectively. Hence \(MSR=8.10/1=8.10\), \(MSE=1.90/3=0.633333\ldots\), and \(F=8.10/0.633333\ldots=12.78947\ldots\). Rounding only the final F statistic to three decimals gives \(12.789\).
Common mistake avoided: Using the response total degrees of freedom, 4, as the residual degrees of freedom would give the wrong \(MSE\). The residual degrees of freedom account for estimating both the intercept and the one slope, so they are \(5-2=3\).
Worked Example: A Multiple-Regression ANOVA Table
A hypothetical pricing study fits an intercept model with two predictors to \(n=8\) observations. Its centered total sum of squares is \(120\) and its residual sum of squares is \(30\), both in squared response units. Construct the table and calculate the overall F statistic.
Method. Recover the regression sum of squares from the decomposition, then calculate the degrees of freedom and mean squares.
The regression sum of squares is \(SSR=SST-SSE=120-30=90\), and the check is \(90+30=120\). With \(p=2\), the regression degrees of freedom are 2, the residual degrees of freedom are \(8-2-1=5\), and the total degrees of freedom are \(8-1=7\). The degrees of freedom check is \(2+5=7\).
The mean squares are \(MSR=90/2=45\) and \(MSE=30/5=6\). Thus \(F=45/6=7.5\). The numerator and denominator degrees of freedom for the overall test are 2 and 5.
Common mistake avoided: The regression degrees of freedom equal the number of slopes tested jointly, here two. The intercept is fitted but is not one of the slopes in the overall null hypothesis.
Worked Example: Recovering Missing Sums of Squares
A hypothetical regression ANOVA summary has \(n=12\), three predictors, \(MSR=36\), and \(MSE=4.5\). Fill in the degrees of freedom, sums of squares, and F statistic, and check the table.
Method. Use the model size to find the degrees of freedom, multiply each mean square by its degrees of freedom to recover its sum of squares, and then check the additive identities.
The regression degrees of freedom are \(p=3\). The residual degrees of freedom are \(12-3-1=8\), and the total degrees of freedom are \(12-1=11\). Therefore \(SSR=MSR\times 3=36\times3=108\), and \(SSE=MSE\times8=4.5\times8=36\). The total sum of squares is \(SST=108+36=144\). The degrees of freedom also add: \(3+8=11\).
Finally, \(F=MSR/MSE=36/4.5=8\). The completed entries are regression \((df=3, SS=108, MS=36, F=8)\), residual \((df=8, SS=36, MS=4.5)\), and total \((df=11, SS=144)\).
Common mistake avoided: A mean square is not itself a sum of squares. To recover a sum of squares, multiply the mean square by its row's degrees of freedom.
Reading and Checking the Table
A reliable construction follows a fixed order. Start with \(n\) and the number \(p\) of predictors, confirm that the fitted model includes an intercept, and identify which sums of squares or mean squares are supplied. Then complete the degrees of freedom, calculate any missing sum of squares, and compute the mean squares and F statistic. Finish by checking both the degrees-of-freedom and sums-of-squares identities.
Record the sample size \(n\), number of slope predictors \(p\), and whether the model includes an intercept.
For an intercept model, use \(p\) for regression, \(n-p-1\) for residual, and \(n-1\) for total.
Use \(SST=SSR+SSE\) to find a missing centered sum of squares.
Divide each model or error sum of squares by its own degrees of freedom, then calculate \(F=MSR/MSE\).
Verify \(p+(n-p-1)=n-1\) and \(SSR+SSE=SST\). An inconsistency signals an arithmetic or transcription error.
The mean squares are on the squared-response scale. The F statistic is a ratio of mean squares and is therefore unitless. The residual mean square estimates the common error variance under the model assumptions; it supplies the denominator for the overall F test. Exact F-test conclusions require the stated assumptions, particularly independent normal errors with constant variance. A correctly assembled table does not by itself establish that those assumptions hold.
A frequent exam trap is to confuse this overall test with a test of one coefficient. The overall test compares all \(p\) slopes jointly with zero and uses \(p\) numerator degrees of freedom. A test of one slope has a different hypothesis and does not use the regression row in the same way. Also check whether a question asks for a sum of squares or a mean square: the latter has already been divided by degrees of freedom.
Check Your Understanding
For each problem, construct or complete the regression ANOVA entries. Assume an ordinary least squares model with an intercept unless stated otherwise.
- A simple regression has \(n=9\), \(SST=64\), and \(SSE=24\). Find \(SSR\), all three degrees of freedom, both mean squares, and the overall F statistic. (Answer: \(SSR=40\); degrees of freedom \(1,7,8\); \(MSR=40\), \(MSE=24/7\approx3.4286\), \(F\approx11.6667\).)
- A model with three predictors has \(n=15\), \(SST=210\), and \(SSE=90\). Find the remaining sums of squares, degrees of freedom, and F statistic. (Answer: \(SSR=120\); degrees of freedom \(3,11,14\); \(MSR=40\), \(MSE=90/11\approx8.1818\), \(F\approx4.8889\).)
- A regression with two predictors and \(n=10\) has \(SSR=48\) and \(SSE=32\). Complete the table and check that its sums of squares and degrees of freedom add correctly. (Answer: \(SST=80\); degrees of freedom \(2,7,9\); \(MSR=24\), \(MSE=32/7\approx4.5714\), \(F=5.25\); both totals check.)
- A regression with four predictors and \(n=20\) has \(MSR=15\) and \(MSE=2.5\). Recover \(SSR\), \(SSE\), \(SST\), the degrees of freedom, and \(F\). (Answer: \(SSR=60\), \(SSE=37.5\), \(SST=97.5\); degrees of freedom \(4,15,19\); \(F=6\).)