Why Regression Coefficient Statistics Follow t Distributions

Why a t Distribution Appears in Regression

A regression coefficient is estimated from a sample, so the estimate varies from sample to sample. If the error variance were known, a coefficient estimate could be standardized using that known variance. In practice, the variance is usually unknown and is estimated from the regression residuals. That substitution changes the standardized statistic: under the normal-error model, it follows a t distribution, not a standard normal distribution.

The reason is a specific combination of results. A centered regression coefficient estimate is normally distributed. Separately, the residual sum of squares, after division by the true error variance, has a chi-square distribution with \(n-2\) degrees of freedom. Under normal errors these two quantities are independent. Their ratio, with the chi-square quantity providing the estimated variance in the denominator, has a t distribution.

Write \(\bar{x}\) for the mean of the predictor values and \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). Let \(b_0\) and \(b_1\) be the least-squares estimates of \(\beta_0\) and \(\beta_1\). The residual for observation \(i\) is \(e_i=Y_i-(b_0+b_1x_i)\), and the residual sum of squares is \(\mathrm{SSE}=\sum_{i=1}^n e_i^2\). Because two regression coefficients have been estimated, the residual variance estimate uses \(n-2\) degrees of freedom: \(s^2=\mathrm{SSE}/(n-2)\).

The Coefficient Standard Errors and Their t Statistics

Under the stated model, the sampling variances of the least-squares estimates are \(\operatorname{Var}(b_1)=\sigma^2/S_{xx}\) and \(\operatorname{Var}(b_0)=\sigma^2(1/n+\bar{x}^2/S_{xx})\). Replacing \(\sigma\) by the residual standard deviation \(s\) gives the estimated standard errors. Thus the slope and intercept statistics, centered at their true coefficients, are

Each statistic has a \(t\) distribution with \(n-2\) degrees of freedom. More generally, the same result applies to a linear combination of the regression coefficients when its standard error is formed using \(s\) and the model-based variance of that combination. For a statistic centered at a proposed value instead of the true coefficient, the exact \(t\) statement holds when the proposed value equals the true coefficient.

Why the Ratio Has a t Distribution

The derivation makes clear why estimating the error variance matters. Let \(Y\) be the vector of responses, let \(X\) be the \(n\)-by-2 matrix whose rows are \((1,x_i)\), and let \(\boldsymbol{\beta}=(\beta_0,\beta_1)^\mathsf{T}\). The model can be written \(Y=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\). The least-squares estimate is \(\hat{\boldsymbol{\beta}}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\). Since it is a linear transformation of a normal error vector, \(\hat{\boldsymbol{\beta}}\) is normal with mean \(\boldsymbol{\beta}\) and covariance matrix \(\sigma^2(X^\mathsf{T}X)^{-1}\).

Let \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) be the projection matrix onto the fitted regression space. The residual vector is \((I-H)Y=(I-H)\boldsymbol{\varepsilon}\), because \((I-H)X=0\). The matrix \(I-H\) is symmetric and idempotent, and its rank is \(n-2\). A standard normal quadratic-form result therefore gives the residual sum of squares distribution shown below.

The coefficient estimate depends on the projection of the errors into the fitted space, while the residual vector depends on the projection into the orthogonal residual space. These projections have zero covariance: \(X^\mathsf{T}(I-H)=0\). They are jointly normal because they are linear transformations of the normal error vector. Jointly normal quantities with zero covariance are independent, so the coefficient estimate is independent of \(\mathrm{SSE}\), and hence of \(s^2\).

For either coefficient, its centered estimate divided by its true standard deviation is a standard normal variable \(Z\). The independent variance estimate satisfies \(s^2/\sigma^2=(\mathrm{SSE}/\sigma^2)/(n-2)\). Consequently, the coefficient statistic is the ratio of \(Z\) to the square root of an independent chi-square variable divided by its degrees of freedom. That ratio is the definition of a Student t random variable.

Worked Examples

What the t Result Does—and Does Not—Say

The t distribution accounts for uncertainty from estimating \(\sigma^2\). Its heavier tails, compared with a standard normal distribution, reflect that additional uncertainty. With more residual degrees of freedom, the t distribution approaches the standard normal distribution, but for a finite sample the exact result is \(t_{n-2}\) under the stated assumptions.

The exact result is conditional on the predictor values and depends on the normal-error model. If errors are not normal, the coefficient may still have useful large-sample approximations under suitable conditions, but the exact independence-and-ratio derivation above no longer establishes a t distribution. If the error variance is known rather than estimated, the denominator does not contain the independent chi-square estimate, and the corresponding standardized coefficient is normal instead.

The central idea is not that any ratio involving a regression estimate automatically has a t distribution. The exact result follows because the model makes the centered coefficient normal, makes the scaled residual sum of squares chi-square, and makes these quantities independent. Knowing which assumptions create those three properties helps identify both when the result applies and when it does not.