Deriving the Regression Coefficient T Test

Why a Regression Coefficient Test Uses a t Statistic

A regression coefficient estimate is uncertain because it is calculated from a sample. To test a claim about a coefficient, we compare the estimate with the value specified by the null hypothesis, measured in estimated standard errors. The resulting statistic has a t distribution rather than a standard normal distribution because the error variance is usually unknown and must itself be estimated from the regression residuals.

The key is that this is not merely a convenient adjustment. Under the usual normal-error assumptions, the standardized coefficient estimate and the residual-based variance estimate have particular distributions and are independent. Their ratio gives the t statistic, with degrees of freedom determined by how many residual degrees of freedom remain after fitting the model.

Model and Assumptions

Write a multiple linear regression model in matrix form as \(Y=X\beta+\varepsilon\). Here \(Y\) is the \(n\)-by-1 vector of response values, \(X\) is the \(n\)-by-\(p\) design matrix, and \(\beta\) is the \(p\)-by-1 vector of regression coefficients. The \(p\) columns of \(X\) represent the model terms, including an intercept if one is included.

Assumptions for the exact coefficient t test: Conditional on the observed design matrix \(X\), the model is \(Y=X\beta+\varepsilon\), where the errors are independent and normally distributed with mean zero and common variance \(\sigma^2\). The columns of \(X\) are linearly independent, so \(X\) has full column rank \(p\). The tested coefficient is one of the coefficients in this model.

The normality, common-variance, and independence assumptions describe the errors around the model, not the distribution of the observed predictor values. Full column rank ensures that the coefficients can be estimated uniquely. The exact t result below is conditional on \(X\); in practice, the assumptions need to be reasonable for the data and model being analyzed.

Define \(C=(X^\mathsf{T}X)^{-1}\). The ordinary least-squares estimate is \(\hat{\beta}=CX^\mathsf{T}Y\). The residual sum of squares is \(SSE=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\), and the estimate of the error variance is \(s^2=SSE/(n-p)\). Thus \(s=\sqrt{SSE/(n-p)}\) estimates the error standard deviation \(\sigma\). The residual degrees of freedom are \(df=n-p\).

From the Coefficient Estimate to Its Standard Error

Let \(\beta_j\) be one coefficient and let \(\hat{\beta}_j\) be its least-squares estimate. The diagonal entry of \(C\) corresponding to that coefficient is written \(c_{jj}\). Under the model assumptions, the estimate has conditional variance \(\sigma^2c_{jj}\). Replacing the unknown \(\sigma\) by its estimate \(s\) gives the estimated standard error \(SE(\hat{\beta}_j)=s\sqrt{c_{jj}}\).

Definition: For testing \(H_0:\beta_j=\beta_{j,0}\), the regression coefficient t statistic is

t=\frac{\hat{\beta}_j-\beta_{j,0}}{s\sqrt{c_{jj}}}.

The quantity \(s\sqrt{c_{jj}}\) is the estimated standard error of \(\hat{\beta}_j\). When the assumptions above hold and \(H_0\) is true, the statistic has a \(t\) distribution with \(n-p\) degrees of freedom.

The standard error has the same units as the coefficient, so the numerator and denominator are in matching units and \(t\) is unitless. For a slope with outcome units per predictor unit, \(\sigma\) has outcome units and \(c_{jj}\) has inverse-predictor-units squared. In particular, \(c_{jj}\) is not measured in predictor-units squared.

Why the Reference Distribution Is t

The derivation uses two ingredients: a normally distributed coefficient estimate when its null value is correct, and an independent estimate of error variance based on residuals. Their ratio is what creates the t distribution.

Standardize the coefficient estimate.
Since \(\hat{\beta}=CX^\mathsf{T}Y=\beta+CX^\mathsf{T}\varepsilon\), it follows that \(\hat{\beta}\) is normally distributed conditional on \(X\), with mean \(\beta\) and covariance matrix \(\sigma^2C\). Under \(H_0:\beta_j=\beta_{j,0}\), this gives \((\hat{\beta}_j-\beta_{j,0})/(\sigma\sqrt{c_{jj}})\sim N(0,1)\).

Express the residual sum of squares as a projection.
Let \(H=XCX^\mathsf{T}\) and \(M=I-H\). The residual vector is \(M Y=M\varepsilon\), because \(MX=0\). Thus \(SSE=\varepsilon^\mathsf{T}M\varepsilon\). The matrix \(M\) is symmetric and idempotent, and its rank is \(n-p\).

Obtain the residual variance distribution.
Under normal errors, an orthogonal change of coordinates diagonalizes the projection \(M\): it retains \(n-p\) mutually independent standard-normal error components and removes the other \(p\). Therefore \(SSE/\sigma^2\) is the sum of squares of \(n-p\) independent standard normal variables, so \(SSE/\sigma^2\sim\chi^2_{n-p}\).

Use independence to form the ratio.
The coefficient estimate depends on the projection of \(\varepsilon\) onto the column space of \(X\), while the residuals depend on its orthogonal complement. These projections are independent: they are jointly normal and have covariance zero. Thus the standardized coefficient estimate and \(SSE/\sigma^2\) are independent.

For completeness, if \(Z\sim N(0,1)\) and \(Q\sim\chi^2_{\nu}\) are independent, then \(Z/\sqrt{Q/\nu}\) has a \(t\) distribution with \(\nu\) degrees of freedom. In the regression derivation, \(Z=(\hat{\beta}_j-\beta_{j,0})/(\sigma\sqrt{c_{jj}})\), \(Q=SSE/\sigma^2\), and \(\nu=n-p\). Since \(s^2=SSE/(n-p)\), substitution gives

\frac{Z}{\sqrt{Q/(n-p)}}= \frac{\hat{\beta}_j-\beta_{j,0}}{\sigma\sqrt{c_{jj}}} \sqrt{\frac{(n-p)\sigma^2}{SSE}} = \frac{\hat{\beta}_j-\beta_{j,0}}{s\sqrt{c_{jj}}} =t.

This establishes the reference distribution under the null: \(t\sim t_{n-p}\). Estimating \(\sigma\) adds uncertainty relative to knowing it exactly; the t distribution accounts for that uncertainty. Its degrees of freedom come from the residual variance estimate, not from the number of coefficients being tested.

Worked Coefficient Tests

Worked Example: A Slope Test With Three Model Coefficients

Suppose a hypothetical regression predicts examination score in points using study hours and a second predictor. There are \(n=12\) observations and \(p=3\) coefficients: an intercept and two slopes. For the study-hours slope, suppose the fitted estimate is \(2.40\) points per hour, \(SSE=45.0\) points squared, and \(c_{jj}=0.0400\) hour\(^{-2}\). Test the coefficient against zero by calculating its t statistic.

The residual degrees of freedom are \(df=n-p=12-3=9\). The estimated residual variance and residual standard deviation are

s^2=\frac{45.0}{9}=5.00\ \text{points}^2, \qquad s=\sqrt{5.00}=2.236\ \text{points}.

The slope standard error is

SE(\hat{\beta}_j)=s\sqrt{c_{jj}} =2.236\sqrt{0.0400} =2.236(0.2000) =0.4472\ \text{points per hour}.

For \(H_0:\beta_j=0\), the statistic is

t=\frac{2.40-0}{0.4472}=5.367.

Under the stated normal-error assumptions and this null hypothesis, the statistic follows a \(t\) distribution with 9 degrees of freedom. The positive value says the fitted slope is 5.367 estimated standard errors above zero. This calculation alone is the test statistic; a decision or p-value also requires specifying the alternative and significance level.

Worked Example: Testing Against a Nonzero Benchmark

In a hypothetical model of recovery time in days, a coefficient for a treatment-score predictor is estimated as \(-0.60\) days per score point. Suppose \(n=15\), \(p=4\), \(SSE=72.0\) days squared, and \(c_{jj}=0.0250\) score-points\(^{-2}\). Calculate the statistic for \(H_0:\beta_j=-1.00\) day per score point.

There are \(15-4=11\) residual degrees of freedom. The variance estimate is \(s^2=72.0/11=6.545\) days squared, so \(s=\sqrt{6.545}=2.558\) days. Then

SE(\hat{\beta}_j)=\sqrt{\frac{72.0}{11}(0.0250)} =0.404520\ldots \approx0.4045\ \text{days per score point}.

The numerator compares the estimate with the null benchmark, not automatically with zero:

t=\frac{-0.60-(-1.00)}{\sqrt{\frac{72.0}{11}(0.0250)}} =\frac{0.40}{\sqrt{\frac{72.0}{11}(0.0250)}} =0.9888.

Under this null and the model assumptions, the statistic has a \(t\) distribution with 11 degrees of freedom. The estimate is 0.9888 estimated standard errors above the benchmark. The calculation does not by itself establish whether the coefficient differs statistically from the benchmark; that conclusion depends on the test's alternative and decision rule.

Worked Example: A Simple Regression Slope

Suppose a hypothetical simple regression relates medication dose in milligrams to a measured response in milliliters. The model has an intercept and one slope, with \(n=10\), \(SSE=32.0\) milliliters squared, fitted slope \(0.750\) milliliters per milligram, and slope multiplier \(c_{jj}=1/30=0.03333\) milligram\(^{-2}\). Find the statistic for a zero-slope null hypothesis.

Here \(p=2\), so \(df=10-2=8\). The residual variance estimate is \(32.0/8=4.00\) milliliters squared and \(s=2.00\) milliliters. The estimated slope standard error is

SE(\hat{\beta}_j)=2.00\sqrt{\frac{1}{30}} =\frac{2.00}{\sqrt{30}} =0.3651\ \text{milliliters per milligram}.

Therefore,

t=\frac{0.750-0}{0.3651}=2.054.

Under the zero-slope null and the assumed model conditions, this statistic has a \(t\) distribution with 8 degrees of freedom. The smaller residual degrees of freedom here, compared with a model with more observations and the same number of fitted terms, affect the reference distribution even though the statistic's formula is unchanged.

Common Pitfalls and Why the Derivation Matters

Using the wrong null value in the numerator. The numerator is \(\hat{\beta}_j-\beta_{j,0}\). It is \(\hat{\beta}_j\) only when the null value is zero.
Using the wrong degrees of freedom. For a model with \(p\) estimated coefficients and \(n\) observations, the residual degrees of freedom are \(n-p\), not \(n-1\) unless \(p=1\).
Confusing variance and standard deviation. The residual variance estimate is \(s^2=SSE/(n-p)\); the coefficient standard error uses \(s\sqrt{c_{jj}}\), not \(s^2\sqrt{c_{jj}}\).
Assuming the t result holds without its conditions. The exact reference distribution depends on independent normal errors with common variance, as well as a full-rank design. If those assumptions are unsuitable, the conventional exact t calibration may not be justified.
Giving \(c_{jj}\) the wrong units. For a slope, its units are inverse predictor-units squared. Multiplying by an error variance in outcome-units squared and taking a square root must produce the coefficient's outcome-per-predictor units.

The derivation matters because it identifies precisely what the reported standard error and reference distribution rely on. The t statistic is not a universal measure detached from the model: its numerator uses the stated null value, its standard error uses the fitted model's residual variation and coefficient variance multiplier, and its degrees of freedom reflect the amount of residual information available.

Check Your Understanding

Use the model assumptions and derivation in this tutorial to answer the following.

A regression has \(n=24\) observations and \(p=5\) coefficients. What are its residual degrees of freedom, and which part of the t-test derivation uses them?
For \(H_0:\beta_j=2\), an estimate is \(2.8\) and its standard error is \(0.4\). Write and calculate the t statistic.
Why does estimating \(\sigma\) from residuals lead to a t statistic rather than a standard normal statistic?
In the projection argument, why are the coefficient estimate and residual sum of squares independent under normal errors?
A slope has units of outcome-units per kilometer. What are the units of its \(c_{jj}\), and what units does \(s\sqrt{c_{jj}}\) have?