Why Degrees of Freedom Matter in a Regression Test
A regression coefficient test compares an observed coefficient estimate with the value specified by the null hypothesis, measured in standard-error units. To judge whether that standardized result is unusual, the test uses a reference distribution. In ordinary least squares, that reference is a Student t distribution whose degrees of freedom come from the residuals: the observations left over after estimating the regression parameters.
The degrees of freedom are therefore connected to the model’s size. Estimating more independent regression coefficients leaves fewer residual degrees of freedom for estimating unexplained variability. This tutorial focuses on how to count those degrees of freedom and how that count determines the reference distribution for a coefficient test.
Counting the Parameters in the Model
A parameter is a population quantity represented by a coefficient in the specified regression model. For example, a simple linear regression has an intercept and a slope, so it estimates two regression parameters. A multiple regression with three predictors and an intercept estimates four regression parameters. The intercept counts even though it is not usually described as a predictor.
In a model with an intercept and \(p\) predictor terms, the usual calculation is:
The subtraction is by the number of independently estimated coefficients, not by the number of variables mentioned in a research question, nor by the number of hypotheses being tested. Testing one coefficient rather than another does not change the fitted model or its residual degrees of freedom.
More generally, write the model in matrix form as \(Y=X\beta+\varepsilon\), where \(X\) is the design matrix: its columns record the model terms, including a column of ones if there is an intercept. If the columns are linearly independent, the number of estimated coefficients is the number of columns of \(X\). If they are not linearly independent, the relevant count is the rank of \(X\), the number of independent columns. The standard formula is then:
A rank-deficient model cannot uniquely estimate every listed coefficient without additional constraints. For a conventional coefficient test, the coefficient being tested must be estimable. In the common full-rank case, counting coefficients gives the same answer as subtracting the number of columns.
Worked Example: A Simple Linear Regression
Suppose a hypothetical analysis uses \(n=18\) observations to relate a measured outcome \(Y\) to one quantitative predictor \(x\), with a straight-line model that includes an intercept. The predictor varies across observations, so the intercept and slope can be estimated separately. Assume the usual regression conditions for an exact coefficient test are appropriate for this constructed example.
The model has two regression coefficients: an intercept and a slope. Thus \(m=2\), and the residual degrees of freedom are:
A test of the slope against a specified null value uses a \(t\) reference distribution with 16 degrees of freedom. The count is not \(18-1=17\), because two coefficients, not one, were estimated to fit the line.
How Residual Degrees of Freedom Set the Reference Distribution
Let \(b_j\) be the least-squares estimate of a regression coefficient \(\beta_j\), and let \(SE(b_j)\) be its estimated standard error. For a test of \(H_0:\beta_j=\beta_{j,0}\), the coefficient test statistic is:
The statistic is compared with a Student t distribution using the regression’s residual degrees of freedom. A t distribution is indexed by its degrees of freedom; smaller degrees of freedom give a distribution with heavier tails, reflecting the additional uncertainty in estimating residual variability from fewer residual degrees of freedom.
The standard error in this statistic uses the residual mean square. If \(e_i\) is the residual for observation \(i\), the residual sum of squares is \(SSE=\sum_{i=1}^{n}e_i^2\), and the estimate of the error variance is:
The t reference distribution arises because the estimated standard error uses this estimated residual variability rather than the unknown \(\sigma\). The degrees of freedom specify the distribution used for that estimate, and therefore the t distribution used to assess the coefficient statistic. This tutorial uses the result to identify the correct degrees of freedom; the derivation of the full coefficient test is a separate topic.
Why the Residual Count Is \(n-m\)
The degrees-of-freedom count can be understood from the geometry of least squares. With \(m\) independent columns in \(X\), the possible fitted-value vectors occupy an \(m\)-dimensional space in the \(n\)-dimensional space of observed outcomes. Least squares chooses the fitted vector in that space closest to the observed outcome vector. The residual vector is perpendicular to the fitted-value space, so it lies in the complementary space, which has dimension \(n-m\).
This dimension is the number of independent residual directions available to measure unexplained variation. Under the normal-error assumptions, the residual sum of squares divided by \(\sigma^2\) has a chi-square distribution with \(n-m\) degrees of freedom. In particular, dividing \(SSE\) by \(n-m\) gives an unbiased estimate of \(\sigma^2\). The exact t result follows from the standard regression result that, under these assumptions, the coefficient estimate and the residual-variance estimate combine to give the stated t reference distribution.
Worked Example: Multiple Regression With Three Predictors
A hypothetical model relates an outcome, such as a laboratory measurement, to three predictors: \(x_1\), \(x_2\), and \(x_3\). It includes an intercept and is fitted to \(n=32\) observations. Suppose the design columns are linearly independent and the regression conditions for an exact coefficient test are assumed to hold.
There are four coefficients: the intercept and one coefficient for each of the three predictors. Therefore \(m=4\), giving:
A test of any one of the three slopes uses a \(t\) distribution with 28 degrees of freedom. The number of tests does not enter this calculation; each test uses the same fitted model and thus the same residual degrees of freedom. The answer would be 28, not \(32-3=29\), because the model also estimates an intercept.
Models With Different Parameterizations
The parameter count follows the independent model terms, not necessarily the number of named groups or the number of coefficients printed in a formula. For example, a four-group model can be written with four group indicators and no intercept, or with an intercept and three group indicators. Each full-rank version has four independently estimated coefficients. The resulting residual degrees of freedom are the same.
Worked Example: Four Group Means Without an Intercept
Imagine a hypothetical data set containing \(n=20\) observations from four groups, with five observations in each group. Consider the model \(Y=\beta_A I_A+\beta_B I_B+\beta_C I_C+\beta_D I_D+\varepsilon\), where each indicator equals 1 for an observation in its named group and 0 otherwise. There is no separate intercept. Because each group is represented, the four indicator columns are independent. Assume the usual error conditions for exact inference.
This model estimates one mean coefficient for each group, for \(m=4\). Its residual degrees of freedom are:
Equivalently, the fitted model uses one parameter for each of the four group means, leaving \(5-1=4\) residual degrees of freedom within each group; across groups, there are \(4\times4=16\). A coefficient test in this model uses \(t_{16}\), provided the tested coefficient is estimable and the model assumptions hold.
If the same model were written with an intercept and all four indicators, the five columns would be linearly dependent: the sum of the four indicators equals the intercept column for every observation. The five coefficients would not be uniquely estimable. One must use an identifiable parameterization or constraints, rather than subtracting five from 20 and claiming that the model has 15 residual degrees of freedom. The fitted-value space still has rank four, so its residual degrees of freedom are \(20-4=16\).
Check the Assumptions and Avoid Common Counting Errors
The arithmetic \(n-m\) gives the residual degrees of freedom for a fitted full-rank model. For those degrees of freedom to specify an exact t reference distribution for a coefficient test, the normal, independent, constant-variance error conditions must also be appropriate, and the coefficient being tested must be estimable. Merely calculating the degrees of freedom does not establish that these conditions hold.
- Using \(n-1\) automatically. That count applies to the usual sample variance around a sample mean. Regression estimates model coefficients; subtract the number of independent regression coefficients instead.
- Forgetting the intercept. In a full-rank model with \(p\) predictors and an intercept, there are \(p+1\) coefficients, so \(df=n-p-1\).
- Subtracting the number of tests. Testing several coefficients does not use up residual degrees of freedom one test at a time. The model fit determines the count.
- Counting redundant columns as separate parameters. If model columns are dependent, use the design-matrix rank. A formula that lists redundant terms does not make them independently estimable.
- Assuming more degrees of freedom guarantee a valid test. The count identifies the reference distribution only when the model and error assumptions for that distribution are appropriate.
A practical check is to list all independently estimated regression coefficients before doing any test calculation. Count the intercept if present, check that the terms are independently estimable, subtract that count from the number of observations, and use the resulting residual degrees of freedom for the coefficient’s t reference distribution.
Check Your Understanding
For each question, identify the regression parameter count before calculating or interpreting the residual degrees of freedom.
- A simple linear regression with an intercept uses 25 observations. How many residual degrees of freedom does a slope test use?
- A multiple regression has five predictors and an intercept, fitted to 40 observations. What is its residual degrees of freedom?
- Why does testing a second coefficient in the same fitted model not change the residual degrees of freedom?
- A four-group model uses four group indicators and no intercept, with all groups represented and 28 observations. How many residual degrees of freedom are there?
- What additional conditions, beyond calculating \(n-m\), are needed for an exact t reference distribution for a coefficient test?