Testing Equality of Two Regression Coefficients

Why Equality Requires a Contrast

A regression output may report separate estimates and standard errors for two coefficients, but the question “Are these coefficients equal?” concerns their difference. The uncertainty in that difference depends not only on the uncertainty in each estimate, but also on how the estimates vary together. In a nonorthogonal design, ignoring that covariance can give the wrong standard error and therefore the wrong test.

Use the normal linear model $Y=X\beta+\varepsilon$, where $X$ is a fixed $n\times p$ design matrix and, conditionally on $X$, $\varepsilon\sim N(0,\sigma^2 I_n)$. Let $r=\operatorname{rank}(X)$. For now, suppose $X$ has full column rank, so the least-squares estimator is $\hat{\beta}=(X^\top X)^{-1}X^\top Y$. The same contrast formulation applies in a rank-deficient model when the contrast is estimable; that qualification is discussed below.

Definition: To test whether coefficients $\beta_j$ and $\beta_k$ are equal, define the contrast $c^\top\beta=\beta_j-\beta_k$, where $c$ has $1$ in position $j$, $-1$ in position $k$, and $0$ elsewhere. The equality hypothesis is $H_0:c^\top\beta=0$. Its estimate is $c^\top\hat{\beta}=\hat{\beta}_j-\hat{\beta}_k$.

The Difference and Its Variance

Under the stated model, $\hat{\beta}$ is unbiased and has covariance matrix $\sigma^2(X^\top X)^{-1}$. Consequently, the variance of the estimated difference is

\operatorname{Var}(\hat{\beta}_j-\hat{\beta}_k) =\sigma^2\left\{[(X^\top X)^{-1}]_{jj} +[(X^\top X)^{-1}]_{kk} -2[(X^\top X)^{-1}]_{jk}\right\}.

The final term is the covariance contribution: $\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k)=\sigma^2[(X^\top X)^{-1}]_{jk}$. Equivalently, the variance is $\operatorname{Var}(\hat{\beta}_j)+\operatorname{Var}(\hat{\beta}_k)-2\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k)$. It is not generally correct to add the two coefficient variances and omit their covariance.

Since $\sigma^2$ is unknown, estimate it by the residual mean square $MSE=SSE/(n-r)$, where $SSE$ is the residual sum of squares. Write $\nu=n-r$ for the residual degrees of freedom. The estimated standard error of the difference is the square root of $MSE$ times the corresponding contrast factor.

Theorem: Suppose $X$ is fixed with full column rank, $Y=X\beta+\varepsilon$, and $\varepsilon\sim N(0,\sigma^2 I_n)$ with $\sigma^2>0$. If $\nu=n-p>0$, then under $H_0:\beta_j-\beta_k=0$, $$ T=\frac{\hat{\beta}_j-\hat{\beta}_k} {\sqrt{MSE\left\{[(X^\top X)^{-1}]_{jj}+[(X^\top X)^{-1}]_{kk}-2[(X^\top X)^{-1}]_{jk}\right\}}} \sim t_\nu. $$

Why the Studentized Statistic Is Exact

The exact calibration follows from the normal-model distribution of a contrast and the independent residual estimate of variance. Here is the argument, including the role of the design.

The numerator is normal.
$\hat{\beta}-\beta=(X^\top X)^{-1}X^\top\varepsilon$ is a linear transformation of a normal vector. Thus $c^\top\hat{\beta}-c^\top\beta$ is normal with mean $0$ and variance $\sigma^2 c^\top(X^\top X)^{-1}c$.

The residual mean square has a chi-square distribution.
Let $H=X(X^\top X)^{-1}X^\top$ be the projection onto the column space of $X$. The residual vector is $(I-H)Y=(I-H)\varepsilon$. Because $I-H$ is an idempotent projection of rank $n-p=\nu$, normal orthogonal-projection theory gives $SSE/\sigma^2\sim\chi^2_\nu$, and hence $\nu MSE/\sigma^2\sim\chi^2_\nu$.

The two quantities are independent.
The contrast estimate depends on $H\varepsilon$, since the fitted part of $\varepsilon$ lies in the column space of $X$. The residual depends on $(I-H)\varepsilon$. These are jointly normal linear transformations, and their cross-covariance is $\sigma^2H(I-H)=0$. For jointly normal vectors, zero cross-covariance implies independence.

Apply the definition of a t variable.
Under $H_0$, divide the contrast estimate by its true standard deviation to obtain $Z\sim N(0,1)$. Independently, $U=\nu MSE/\sigma^2\sim\chi^2_\nu$. The displayed statistic is $Z/\sqrt{U/\nu}$, which by definition has a $t_\nu$ distribution.

This proof also explains why the test is exact rather than an approximation: normal errors give the exact normal and chi-square distributions, and the residual variance estimate is independent of the contrast estimate. The assumptions matter. Without the stated normal model, the same statistic need not have an exact $t$ distribution.

Constructing and Interpreting the Test

For a two-sided alternative, $H_A:\beta_j-\beta_k\ne0$, calculate the observed statistic using the estimated difference and its covariance-adjusted standard error. At significance level $\alpha$, reject $H_0$ when $|t_{\mathrm{obs}}|>t_{1-\alpha/2,\nu}$. The two-sided p-value is $2P(T_\nu\ge |t_{\mathrm{obs}}|)$, with $T_\nu\sim t_\nu$. A one-sided alternative uses the corresponding one-sided tail of the same exact reference distribution.

The hypothesis is a statement about the population regression coefficients, not about whether the two separately reported coefficient tests happen to have the same outcome. Nor does a failure to reject establish that the coefficients are equal; it means the data do not provide sufficient evidence against equality at the chosen level.

Worked Example: An Orthogonal Design

Consider a regression with an intercept and two predictors $x$ and $z$, using five observations. Let $x=(1,-1,0,0,0)^\top$, $z=(0,0,1,-1,0)^\top$, and let the response be $y=10\mathbf{1}+2x+z+q$, where $q=(1,1,1,1,-4)^\top$. The vector $q$ is orthogonal to the intercept and both predictors: its entries sum to zero, $x^\top q=1-1=0$, and $z^\top q=1-1=0$.

Therefore the fitted coefficients are $\hat{\beta}_0=10$, $\hat{\beta}_x=2$, and $\hat{\beta}_z=1$, with residual vector $q$. Its squared length is $1+1+1+1+16=20$. There are $5-3=2$ residual degrees of freedom, so $MSE=20/2=10$.

The centered predictors are orthogonal and each has sum of squares $2$, so the $x,z$ block of $(X^\top X)^{-1}$ is diagonal with entries $1/2$. For the equality test $H_0:\beta_x=\beta_z$, the estimated difference is $2-1=1$, and its estimated variance is $10(1/2+1/2)=10$. Thus $SE=\sqrt{10}$ and $t_{\mathrm{obs}}=1/\sqrt{10}\approx0.316$, with $2$ degrees of freedom. At the two-sided $5\%$ level, the critical value is $t_{0.975,2}\approx4.303$, so this test does not reject equality.

Worked Example: Covariance Changes the Standard Error

Take five observations with an intercept and predictors $x=(1,-1,0,0,0)^\top$, $z=(1,0,-1,0,0)^\top$. Their sums are zero, their squared lengths are both $2$, and $x^\top z=1$. The predictor block of $X^\top X$ is therefore $\begin{pmatrix}2&1\\1&2\end{pmatrix}$, whose inverse is $\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. In particular, the estimated coefficients have covariance $-\sigma^2/3$.

Let $q=(1,1,1,-1.5,-1.5)^\top$. It is orthogonal to the intercept because its entries sum to zero; it is orthogonal to $x$ and $z$ because $q_1=q_2=q_3$. Define $y=10\mathbf{1}+3x+2z+q$. The fitted predictor coefficients are $3$ and $2$, and the residual sum of squares is $3(1^2)+2(1.5^2)=7.5$. With $5-3=2$ residual degrees of freedom, $MSE=7.5/2=3.75$.

For $H_0:\beta_x=\beta_z$, the estimated difference is $1$. Its estimated variance is $3.75\{2/3+2/3-2(-1/3)\}=3.75(2)=7.5$, so $SE=\sqrt{7.5}$ and $t_{\mathrm{obs}}=1/\sqrt{7.5}\approx0.365$. If the covariance were incorrectly omitted, the estimated variance would instead be $3.75(2/3+2/3)=5$, producing the incorrect standard error $\sqrt{5}$. The negative covariance increases the variance of the difference; its effect cannot be inferred from the two marginal standard errors alone.

Estimability and a Common Pitfall

In a rank-deficient design, individual coefficients may not be uniquely determined by the distribution of $Y$. A linear function $c^\top\beta$ is estimable when it has the same value for any two parameter vectors $\beta$ and $\beta'$ satisfying $X\beta=X\beta'$. Equivalently, $c$ must lie in the row space of $X$. An exact contrast test requires the coefficient difference to be estimable; otherwise the data cannot identify the difference, so there is no valid contrast standard error or exact t test for it.

Worked Example: Duplicate Predictors Make Their Difference Nonestimable

Suppose a model includes an intercept and two predictors whose observed columns are identical: $x=z$. Then $X\beta$ depends on their coefficients only through $(\beta_x+\beta_z)x$. For any number $a$, replacing $(\beta_x,\beta_z)$ by $(\beta_x+a,\beta_z-a)$ leaves this sum unchanged and hence leaves $X\beta$ unchanged.

But the difference changes from $\beta_x-\beta_z$ to $(\beta_x-\beta_z)+2a$. Since the same mean response can correspond to different values of $\beta_x-\beta_z$, that difference is not estimable. A software package may drop one duplicate column or report a singularity; treating the resulting output as though both coefficients had separately identified estimates does not create a test of their equality.

A common mistake is to calculate $SE(\hat{\beta}_j-\hat{\beta}_k)$ as $\sqrt{SE(\hat{\beta}_j)^2+SE(\hat{\beta}_k)^2}$. That expression is correct only when the two estimates have covariance zero. The correct variance always includes $-2\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k)$; in a full-rank normal linear model it is obtained directly from $MSE\,c^\top(X^\top X)^{-1}c$. Another mistake is to test the two coefficients separately and infer equality from those two decisions. Equality is one contrast hypothesis and should be tested as such.

Key takeaway: Test $H_0:\beta_j=\beta_k$ by estimating the contrast $\hat{\beta}_j-\hat{\beta}_k$, using its full covariance-adjusted standard error, and comparing the resulting statistic with the exact $t_{n-r}$ distribution under the Gaussian spherical-error model. First verify that the contrast is estimable.

Check Your Understanding

Use the contrast formulation and exact-test conditions to answer these questions.

Write the contrast vector for testing $\beta_2=\beta_4$ in a model with five coefficients, including the intercept.
If $\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k)$ is positive, how does including it affect the variance of their estimated difference?
A model has $n=30$ observations and design rank $r=6$. What are the residual degrees of freedom for the exact contrast t test?
Why does the normal linear model give independence between a coefficient contrast estimate and the residual sum of squares?
What feature of duplicate predictor columns prevents a test of the difference between their coefficients?