Exact Tests for a Single Regression Coefficient

Why Test a Single Regression Coefficient?

A regression coefficient describes how the fitted mean changes with one model component while the other components are held fixed. Testing whether that coefficient is zero asks whether the corresponding component has a zero coefficient in the specified model. In a normal linear model, the answer can be based on an exact t distribution, not just a large-sample approximation, provided the model assumptions hold.

The test is specific to the model and design matrix being used. In particular, the coefficient’s standard error reflects the other columns of the design matrix, and its degrees of freedom come from estimating all \(p\) coefficients. The test does not, by itself, establish causation or answer whether the model is appropriate.

Model, Hypothesis, and Test Statistic

Definition: Consider the fixed-design normal linear model \(Y=X\beta+\varepsilon\), where \(Y\) is an \(n\)-vector, \(X\) is a known \(n\times p\) matrix of rank \(p\), and \(\beta=(\beta_1,\ldots,\beta_p)'\). Assume \(n>p\) and \(\varepsilon\sim N_n(0,\sigma^2 I_n)\), with \(\sigma^2>0\). For a specified coefficient \(\beta_j\), the usual two-sided single-coefficient test is \(H_0:\beta_j=0\) against \(H_1:\beta_j\ne 0\).

The least-squares estimator is \(\hat{\beta}=(X'X)^{-1}X'Y\). Write \(c_{jj}\) for the \(j\)th diagonal entry of \((X'X)^{-1}\). The residual sum of squares and residual mean square are

\mathrm{SSE}=(Y-X\hat{\beta})'(Y-X\hat{\beta}),\qquad s^2=\frac{\mathrm{SSE}}{n-p}.

Here \(s=\sqrt{s^2}\). The estimated standard error of \(\hat{\beta}_j\) is \(s\sqrt{c_{jj}}\). Thus the observed test statistic is

t_{\mathrm{obs}}=\frac{\hat{\beta}_j}{s\sqrt{c_{jj}}}.

Under \(H_0\), this statistic has a Student t distribution with \(n-p\) degrees of freedom. For a chosen significance level \(\alpha\), the two-sided rule rejects when \(|t_{\mathrm{obs}}|>t_{n-p,\,1-\alpha/2}\), where \(t_{\nu,\,q}\) denotes the \(q\) quantile of the t distribution with \(\nu\) degrees of freedom. The corresponding p-value is

p\text{-value}=P\!\left(|T|\ge |t_{\mathrm{obs}}|\right), \qquad T\sim t_{n-p}.

This is the probability, calculated under the null model, of a statistic at least as far from zero as the observed one. It is not the probability that \(H_0\) is true.

Why the t Distribution Is Exact

The exact result combines the sampling distribution of the coefficient estimate with the independent estimate of error variance. The fixed-design and Gaussian assumptions matter: they give a normal coefficient estimate and an independent chi-square residual sum of squares.

Theorem: Under the model assumptions above, if \(H_0:\beta_j=0\) is true, then \[ \frac{\hat{\beta}_j}{s\sqrt{c_{jj}}}\sim t_{n-p}. \] Consequently, the two-sided test using the \(t_{n-p}\) critical value has rejection probability exactly \(\alpha\) under the null.

Proof. Define \(P_X=X(X'X)^{-1}X'\) and \(M_X=I_n-P_X\). Because \(X\) has rank \(p\), \(P_X\) is a symmetric idempotent projection of rank \(p\), and \(M_X\) is a symmetric idempotent projection of rank \(n-p\). Also, \(\hat{\beta}=\beta+(X'X)^{-1}X'\varepsilon\). Therefore,

\hat{\beta}_j\sim N(\beta_j,\sigma^2 c_{jj}).

The residual vector is \(M_XY=M_X\varepsilon\), since \(M_XX=0\). The normal-theory result for a symmetric idempotent projection gives

\frac{\mathrm{SSE}}{\sigma^2} =\frac{\varepsilon'M_X\varepsilon}{\sigma^2} \sim\chi^2_{n-p}.

Moreover, the coefficient estimate and residual vector are independent: both are linear transformations of the jointly normal vector \(\varepsilon\), and their cross-covariance is zero because \(X'M_X=0\). Hence \(\hat{\beta}_j\) and \(\mathrm{SSE}\) are independent. Under \(H_0\), \(Z=\hat{\beta}_j/(\sigma\sqrt{c_{jj}})\sim N(0,1)\), independently of \(U=\mathrm{SSE}/\sigma^2\sim\chi^2_{n-p}\). Since \(s^2=\mathrm{SSE}/(n-p)\),

\frac{\hat{\beta}_j}{s\sqrt{c_{jj}}} =\frac{Z}{\sqrt{U/(n-p)}}\sim t_{n-p}.

The t distribution is continuous and symmetric, so the probability that its absolute value exceeds \(t_{n-p,\,1-\alpha/2}\) is exactly \(\alpha\). This proves the stated distribution and exact size. \(\square\)

The proof also explains the degrees of freedom: estimating the \(p\) fitted coefficients leaves \(n-p\) residual degrees of freedom. The entry \(c_{jj}\) accounts for the design and for how the \(j\)th column relates to the other columns; it is not generally the reciprocal of that column’s sum of squares unless additional design conditions make it so.

Worked Tests

Worked Example: A Coefficient in a Multiple Regression

Let \(n=6\), with design columns \(1\), \(x=(-1,-1,0,0,1,1)'\), and \(z=(1,-1,1,-1,0,0)'\). These columns are mutually orthogonal, with squared lengths \(6,4,4\), respectively. Suppose the observed response is \(y=2\cdot 1+x+0.5z+r\), where \(r=(1,-1,-1,1,0,0)'\). The vector \(r\) is orthogonal to all three design columns, and \(r'r=4\).

Orthogonality gives the fitted coefficients directly: the intercept is \(2\), the coefficient of \(x\) is \(1\), and the coefficient of \(z\) is \(0.5\). The residual is \(r\), so \(\mathrm{SSE}=4\). There are \(p=3\) coefficients; hence the residual degrees of freedom are \(6-3=3\), and \(s^2=4/3\).

The diagonal entry for the \(x\) coefficient is \(c_{xx}=1/(x'x)=1/4\). Its estimated standard error is therefore \[ s\sqrt{c_{xx}}=\sqrt{4/3}\sqrt{1/4}=\frac{1}{\sqrt{3}}. \] The test statistic for \(H_0:\beta_x=0\) is \(t_{\mathrm{obs}}=1/(1/\sqrt{3})=\sqrt{3}\approx1.732\). With three degrees of freedom, the two-sided p-value is approximately \(0.182\). At level \(0.05\), the critical value is \(t_{3,\,0.975}\approx3.182\), and \(|1.732|<3.182\). The test does not reject \(H_0\) at that level.

Worked Example: A Small Estimated Slope

In a simple regression with an intercept, let \(x=(-2,-1,0,1,2)'\), \(r=(1,-2,2,-2,1)'\), and \(y=3\cdot 1+0.4x+r\). Both \(x\) and \(r\) sum to zero, and \(x'r=0\). Thus the fitted intercept is \(3\), the fitted slope is \(0.4\), and the residual vector is \(r\).

Here \(n=5\), \(p=2\), and \(\mathrm{SSE}=r'r=1+4+4+4+1=14\). So the residual degrees of freedom are \(3\) and \(s^2=14/3\). Since \(x'x=10\), the diagonal entry for the slope is \(1/10\), giving \[ \operatorname{SE}(\hat{\beta}_{\text{slope}}) =\sqrt{\frac{14}{3}\cdot\frac{1}{10}} =\sqrt{\frac{7}{15}}\approx0.683. \] The statistic is \(0.4/0.683\approx0.586\). Its two-sided p-value from \(t_3\) is approximately \(0.599\). This is not evidence to reject a zero slope at level \(0.05\); it does not prove that the population coefficient is zero.

Worked Example: A Coefficient with a Small p-Value

Use the six-observation design from the first example, but let \(y=2\cdot1+2x+0.5z+0.5r\), with the same mutually orthogonal vectors \(x,z,r\). The fitted coefficient of \(x\) is \(2\), the residual is \(0.5r\), and \(\mathrm{SSE}=0.5^2(4)=1\). As before, \(n-p=6-3=3\), so \(s^2=1/3\), and \(c_{xx}=1/4\).

The estimated standard error is \(\sqrt{(1/3)(1/4)}=1/\sqrt{12}\approx0.289\), so \(t_{\mathrm{obs}}=2/(1/\sqrt{12})=4\sqrt{3}\approx6.928\). The two-sided p-value from \(t_3\) is approximately \(0.0062\), below \(0.05\). The test rejects \(H_0:\beta_x=0\) at the five-percent level, conditional on the stated model assumptions.

Common Mistakes and Interpretation

Using \(n-1\) degrees of freedom. The residual degrees of freedom are \(n-p\), where \(p\) is the number of fitted coefficients, including an intercept when present.
Using the standard deviation estimate alone. The coefficient standard error is \(s\sqrt{c_{jj}}\), not merely \(s\). The design-based factor \(c_{jj}\) is essential.
Using a normal reference distribution for an exact small-sample test. With unknown \(\sigma^2\), the exact null reference distribution is \(t_{n-p}\). Replacing it with a standard normal distribution is an approximation, not this exact test.
Interpreting a large p-value as proof of a zero effect. It means the observed statistic is not unusual enough under the null to reject at the chosen level. It does not establish that the null is true.
Ignoring the model assumptions. The finite-sample exactness stated here depends on a full-rank fixed design and independent normal errors with common variance. If those assumptions do not hold, this derivation does not guarantee the stated t distribution.

Key takeaway: For a single coefficient in a full-rank normal linear model, divide its estimate by \(s\sqrt{c_{jj}}\), use \(n-p\) degrees of freedom, and calculate the p-value from the corresponding t distribution. Under the null and stated assumptions, this procedure is exact.

Check Your Understanding

Use the model assumptions and definitions above to answer the following.

A model has \(n=24\) observations and \(p=5\) fitted coefficients. What degrees of freedom should be used for a single-coefficient t test?
Why does the standard error of \(\hat{\beta}_j\) include the factor \(\sqrt{c_{jj}}\) as well as \(s\)?
In the six-observation worked example, what is the two-sided null hypothesis, and why is the reference distribution \(t_3\)?
What does a two-sided p-value of \(0.04\) mean, and what does it not mean?
Name two assumptions used to establish the exact finite-sample t distribution.