Deriving the t Test for a Regression Coefficient

Why a Regression Coefficient Has a t Statistic

A fitted regression coefficient is an estimate, not the true population coefficient. To assess how far an estimate lies from a value specified by a null hypothesis, we measure the difference in estimated standard-error units. Because the error variance is usually unknown, the standard error itself is estimated from the regression residuals. Under the usual normal-error regression assumptions, that studentization leads to a t statistic.

The central questions are therefore connected: what quantity should be standardized, how is its standard error estimated, and why does the resulting statistic have a t distribution with a particular number of degrees of freedom? The derivation also identifies when that exact distributional claim does not hold.

Model and assumptions: For \(n\) observations, write the multiple linear regression model as \(Y=X\beta+\varepsilon\), where \(X\) is a fixed \(n\times p\) design matrix of full column rank, \(\beta\) is a \(p\)-vector of regression coefficients, and \(\varepsilon\sim N(0,\sigma^2 I_n)\). Thus the errors are independent, have mean zero, have a common variance \(\sigma^2\), and are normally distributed. The \(p\) columns of \(X\) count all fitted coefficients, including an intercept when one is included.

Full column rank means the columns of \(X\) are linearly independent, so the coefficients are identifiable and \(X^\mathsf{T}X\) is invertible. The ordinary least squares estimator is \(\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\). The residual sum of squares is \(SSE=\sum_{i=1}^n (Y_i-\hat Y_i)^2\), and the estimated error variance is \(s^2=SSE/(n-p)\). For coefficient \(j\), let \(c_{jj}\) be the \(j\)th diagonal entry of \((X^\mathsf{T}X)^{-1}\). Its estimated standard error is \(SE(\hat\beta_j)=\sqrt{s^2c_{jj}}\).

Deriving the Coefficient t Statistic

Consider the null hypothesis that coefficient \(j\) equals a specified value \(\beta_{j,0}\). The estimate’s difference from that value is \(\hat\beta_j-\beta_{j,0}\). If the null is true, its standard deviation is \(\sigma\sqrt{c_{jj}}\), but \(\sigma\) is unknown. Replacing \(\sigma\) with its residual-based estimate \(s\) gives the statistic used for the coefficient test.

t=\frac{\hat\beta_j-\beta_{j,0}}{s\sqrt{c_{jj}}} =\frac{\hat\beta_j-\beta_{j,0}}{SE(\hat\beta_j)}.

This form is not just a convenient scaling. Its exact reference distribution follows from two results under the model assumptions. First, \(\hat\beta\) is normally distributed with mean \(\beta\) and covariance matrix \(\sigma^2(X^\mathsf{T}X)^{-1}\). Second, \(SSE/\sigma^2\) has a chi-square distribution with \(n-p\) degrees of freedom and is independent of \(\hat\beta\). These facts make the standardized coefficient difference a normal quantity divided by the square root of an independent chi-square quantity per degree of freedom.

Standardize using the true error scale.
Under \(H_0:\beta_j=\beta_{j,0}\), the coefficient estimator satisfies \((\hat\beta_j-\beta_{j,0})/(\sigma\sqrt{c_{jj}})\sim N(0,1)\).

Estimate the error variance.
The residual result gives \(SSE/\sigma^2\sim\chi^2_{n-p}\), so \(s^2/\sigma^2=(SSE/\sigma^2)/(n-p)\) is a chi-square variable divided by its degrees of freedom.

Use independence to form the ratio.
The coefficient estimator and residual sum of squares are independent under normal errors. Therefore the normal quantity in Step 1 is independent of the variance estimate in Step 2.

Recognize the t distribution.
A standard normal variable divided by the square root of an independent chi-square variable divided by its degrees of freedom has a Student t distribution. Thus, under \(H_0\), \(t\sim t_{n-p}\).

The degrees of freedom are \(n-p\), not \(n-1\) automatically and not \(n\) minus the number of predictors if that count omits the intercept. Fitting \(p\) linearly independent coefficients uses \(p\) dimensions of the observed response to estimate the mean structure. The residuals remain in the orthogonal complement of the column space of \(X\), which has dimension \(n-p\). Under normal errors, their scaled squared length therefore has a chi-square distribution with \(n-p\) degrees of freedom.

Theorem: Under the fixed-design, full-rank normal linear regression model, for testing \(H_0:\beta_j=\beta_{j,0}\), the statistic \((\hat\beta_j-\beta_{j,0})/[s\sqrt{c_{jj}}]\) has a central Student t distribution with \(n-p\) degrees of freedom when the null hypothesis is true. If the errors are not normal, this exact finite-sample result is not generally guaranteed, even when the coefficient estimator and residual variance formula can still be calculated.

Worked Calculations

Worked Example: Testing a Coefficient with Four Fitted Parameters

A hypothetical insurer models a claim-cost index using four fitted coefficients: an intercept and three predictors. The data set has \(n=20\) observations. For one predictor, the estimated coefficient is \(0.6\), and the null value is \(0.1\). The regression output gives \(SSE=32\) and \(c_{jj}=0.04\). Assume the normal linear regression model stated above. Calculate the coefficient t statistic and its degrees of freedom.

Method. Estimate the error variance, calculate the coefficient’s estimated standard error, and divide the difference from the null value by that standard error.

Step 1: Count the fitted coefficients. The intercept and three predictors give \(p=4\), so the residual degrees of freedom are \(n-p=20-4=16\).

Step 2: Estimate the error variance. \(s^2=SSE/(n-p)=32/16=2\).

Step 3: Calculate the estimated standard error. \(SE(\hat\beta_j)=\sqrt{s^2c_{jj}}=\sqrt{2(0.04)}=\sqrt{0.08}\approx0.282843\).

Step 4: Standardize the coefficient difference. \(t=(0.6-0.1)/\sqrt{0.08}=0.5/0.282843\approx1.768\). The statistic has 16 residual degrees of freedom; under the stated assumptions and null, it has a central \(t_{16}\) distribution.

Check and common mistake. The numerator is \(0.5\), and the denominator is about \(0.283\), so the quotient must be about \(1.77\). Using \(0.2\) as the standard error would give \(2.5\), but that value does not follow from \(s^2=2\) and \(c_{jj}=0.04\). The correct result is \(t\approx1.768\), not \(2.500\).

Worked Example: Keeping the Null Value in the Numerator

In a hypothetical model of monthly equipment-repair costs, \(n=18\) records are fitted with an intercept and two predictors. For one predictor, \(\hat\beta_j=0.74\), the specified null value is \(\beta_{j,0}=0.14\), \(SSE=45\), and \(c_{jj}=0.025\). Assume independent normal errors with a common variance. Find the t statistic and its degrees of freedom.

Method. Use the estimated variance from the residual sum of squares and standardize the estimate’s difference from the specified null value.

Step 1: Determine the residual degrees of freedom. There are \(p=3\) coefficients, including the intercept, so \(18-3=15\) residual degrees of freedom.

Step 2: Compute the error-variance estimate. \(s^2=45/15=3\).

Step 3: Compute the estimated standard error. \(SE(\hat\beta_j)=\sqrt{3(0.025)}=\sqrt{0.075}\approx0.273861\).

Step 4: Calculate the statistic. The numerator is \(0.74-0.14=0.60\), so \(t=0.60/0.273861\approx2.191\). Its degrees of freedom are 15. Under the null and the stated assumptions, the reference distribution is central \(t_{15}\).

Check and common mistake. The coefficient is being compared with \(0.14\), not with zero. Substituting zero would produce \(0.74/0.273861\), which tests a different null hypothesis. The requested statistic is approximately \(2.191\).

Worked Example: Counting the Intercept and Checking Normality

A hypothetical research team fits a line with an intercept and one slope to \(n=14\) observations of a laboratory response. The slope estimate is \(-0.2\), the null slope is \(-0.8\), \(SSE=24\), and the slope’s diagonal element \(c_{jj}\) is \(0.125\). Calculate the statistic and state what can be concluded about its exact distribution if the errors are independent and have common variance but are not normally distributed.

Method. Calculate the statistic using the usual residual variance estimate, then check whether the assumptions support an exact t distribution.

Step 1: Count the parameters. The fitted line has \(p=2\) coefficients: an intercept and a slope. Thus \(df=n-p=14-2=12\).

Step 2: Calculate the residual variance estimate. \(s^2=24/12=2\).

Step 3: Calculate the slope standard error. \(SE(\hat\beta_j)=\sqrt{2(0.125)}=\sqrt{0.25}=0.5\).

Step 4: Calculate the statistic. \(t=[-0.2-(-0.8)]/0.5=0.6/0.5=1.2\).

Check and common mistake. Subtracting the negative null value gives \(-0.2+0.8=0.6\), not \(-1.0\). The calculated statistic is \(1.2\) with 12 residual degrees of freedom. Because normality is not assumed, it is not guaranteed to have an exact central \(t_{12}\) distribution in this finite sample. The calculation alone does not establish that distributional result.

When the Exact Result Applies

The exact t result depends on more than the algebra of the statistic. Under the stated regression model, normality supplies the normal distribution of the standardized coefficient estimate and the chi-square distribution of the scaled residual sum of squares. The fixed design and full-rank conditions make the coefficient estimator well-defined; the common variance and independence assumptions are part of the model used for these sampling results.

If errors are nonnormal, the same formula can still be computed when the model is fitted, but its exact finite-sample reference distribution need not be \(t_{n-p}\). In some settings a t approximation may be useful, but it must be justified rather than asserted solely because the statistic has the familiar algebraic form. If the design matrix is rank deficient, \((X^\mathsf{T}X)^{-1}\) does not exist as written, and this coefficient-variance derivation cannot be applied in that form.

Common Exam Traps

Forgetting the hypothesized value. The numerator is \(\hat\beta_j-\beta_{j,0}\). It is \(\hat\beta_j\) only when the null value is zero.
Using the wrong standard error. The denominator is \(\sqrt{s^2c_{jj}}\). First calculate \(s^2=SSE/(n-p)\); do not substitute the residual standard deviation \(s\) alone.
Counting only predictors. The parameter count \(p\) includes the intercept if it is fitted. Use the number of estimated coefficients, not merely the number of named explanatory variables.
Claiming an exact t distribution without normal errors. A computed statistic and its exact reference distribution are separate claims. The exact finite-sample t result here relies on the normal linear regression assumptions.
Using the wrong degrees of freedom for the residual variance. The denominator in \(s^2\) is \(n-p\), which is also the degrees of freedom for the coefficient t statistic in this model.

Key takeaway: Under the normal linear regression model, compare a coefficient estimate with its null value, divide by its estimated standard error, and use \(n-p\) degrees of freedom. The exact t distribution follows from the independent normal and chi-square components; outside those assumptions, it is not automatic.

Check Your Understanding

For each problem, calculate the coefficient statistic and its residual degrees of freedom. Treat the exact-distribution question according to the assumptions stated.

A hypothetical pricing model has \(n=16\), three fitted coefficients including the intercept, \(SSE=39\), \(\hat\beta_j=1.1\), null value \(0.5\), and \(c_{jj}=0.04\). Assume normal errors. Find \(s^2\), the standard error, and the statistic. (Answer: \(df=13\), \(s^2=3\), \(SE=\sqrt{0.12}\approx0.3464\), \(t\approx1.732\); under the null, central \(t_{13}\).)
A hypothetical model has \(n=25\) and five fitted coefficients, including the intercept. For one coefficient, \(SSE=80\), \(\hat\beta_j=-0.3\), \(\beta_{j,0}=0.2\), and \(c_{jj}=0.0125\). Assume normal errors. Find the statistic and degrees of freedom. (Answer: \(df=20\), \(s^2=4\), \(SE=\sqrt{0.05}\approx0.2236\), \(t\approx-2.236\); under the null, central \(t_{20}\).)
A hypothetical straight-line model has \(n=12\), \(SSE=20\), \(\hat\beta_j=0.9\), null value \(0.4\), and \(c_{jj}=0.05\). The fitted model includes an intercept, and errors are normal. Find the statistic. (Answer: \(p=2\), \(df=10\), \(s^2=2\), \(SE=\sqrt{0.1}\approx0.3162\), \(t\approx1.581\); under the null, central \(t_{10}\).)
A hypothetical model has \(n=30\), four fitted coefficients, \(SSE=52\), \(\hat\beta_j=0.5\), null value \(0.2\), and \(c_{jj}=0.02\). Errors are independent with common variance but are not normally distributed. Calculate the statistic and describe its exact distributional status. (Answer: \(df=26\), \(s^2=2\), \(SE=0.2\), \(t=1.5\); an exact central \(t_{26}\) distribution is not guaranteed.)
A full-rank regression is fitted to \(n=8\) observations using \(p=8\) coefficients. Can the residual-based t statistic be formed using \(s^2=SSE/(n-p)\)? Explain. (Answer: No. \(n-p=0\), so this formula cannot estimate the error variance and there are no residual degrees of freedom for the stated t statistic.)

What You'll Learn