Exact Tests for Arbitrary Linear Contrasts

Why Test a Linear Contrast?

A scientific question often concerns a combination of model parameters rather than one parameter alone. For example, a researcher may want to know whether the sum of two regression coefficients is zero, or whether a linear combination of a treatment effect and an intercept equals a specified value. These are linear contrast questions. Their tests use the same residual variance estimate as coefficient tests, but the standard error must account for the entire combination, including covariance between coefficient estimates.

This tutorial develops the exact t test for one estimable contrast in a fixed-design Gaussian linear model. Exactness here means that, when the null hypothesis is true and the model assumptions hold, the test statistic has a Student t distribution in finite samples—not merely an approximate large-sample distribution.

Model and Contrast

Definition: Consider the fixed-design linear model $Y=X\beta+\varepsilon$, where $Y$ is an $n$-vector of responses, $X$ is a fixed $n$ by $p$ design matrix of rank $r$, and $\beta$ is a $p$-vector of coefficients. Assume $\varepsilon\sim N(0,\sigma^2 I_n)$, with $\sigma^2>0$. A linear contrast is $a^\mathsf{T}\beta$, where $a$ is a specified $p$-vector. It is estimable when $a$ lies in the row space of $X$; equivalently, $a=X^\mathsf{T}b$ for some $n$-vector $b$.

For a rank-deficient design, different least-squares coefficient vectors can represent the same fitted mean. An estimable contrast has the same value for every such representation. Write $\hat\beta=(X^\mathsf{T}X)^+X^\mathsf{T}Y$, where $+$ denotes the Moore–Penrose inverse. Then $a^\mathsf{T}\hat\beta$ is the least-squares estimate of the estimable contrast. The full-rank case is included: when $r=p$, the Moore–Penrose inverse is the ordinary inverse of $X^\mathsf{T}X$.

Let $P_X$ be the orthogonal projection onto the column space of $X$. The residual sum of squares is $\mathrm{SSE}=Y^\mathsf{T}(I-P_X)Y$, and the residual mean square is $\mathrm{MSE}=\mathrm{SSE}/(n-r)$. We require $n-r>0$, so the model has positive residual degrees of freedom.

Result: Contrast estimate and variance. If $a^\mathsf{T}\beta$ is estimable, then $$ E[a^\mathsf{T}\hat\beta]=a^\mathsf{T}\beta, \qquad \operatorname{Var}(a^\mathsf{T}\hat\beta) =\sigma^2 a^\mathsf{T}(X^\mathsf{T}X)^+a. $$ Define $v_a=a^\mathsf{T}(X^\mathsf{T}X)^+a$. For a nonzero estimable contrast, $v_a>0$.

To see why the covariance structure matters, in a full-rank model the variance multiplier expands as $a^\mathsf{T}(X^\mathsf{T}X)^{-1}a$. This includes both the individual coefficient variances and the covariance terms. It is generally not correct to add only the variances of the coefficients in the contrast.

Deriving the Exact t Statistic

For testing $H_0:a^\mathsf{T}\beta=c$, the natural estimate-minus-null numerator is $a^\mathsf{T}\hat\beta-c$. Its standard deviation depends on the unknown $\sigma$, so the test replaces $\sigma^2$ by the residual mean square. The resulting statistic is

T=\frac{a^\mathsf{T}\hat\beta-c} {\sqrt{\mathrm{MSE}\,a^\mathsf{T}(X^\mathsf{T}X)^+a}}.

Theorem: Exact t test for one estimable contrast. Under the fixed-design Gaussian model above, suppose $a^\mathsf{T}\beta$ is estimable and nonzero, and $n-r>0$. Under $H_0:a^\mathsf{T}\beta=c$, $$ T\sim t_{n-r}. $$ Thus the exact reference distribution uses the residual degrees of freedom $n-r$.

Proof. Since the contrast is estimable, write $a=X^\mathsf{T}b$. The projection identities $P_X X=X$ and $X(X^\mathsf{T}X)^+X^\mathsf{T}=P_X$ give $a^\mathsf{T}\hat\beta=b^\mathsf{T}P_XY$. Its expectation is $b^\mathsf{T}P_XX\beta=b^\mathsf{T}X\beta=a^\mathsf{T}\beta$, and its variance is $\sigma^2 b^\mathsf{T}P_Xb=\sigma^2v_a$. Because it is a linear function of a Gaussian vector, under the null $$ Z=\frac{a^\mathsf{T}\hat\beta-c}{\sigma\sqrt{v_a}}\sim N(0,1). $$ Also, $\mathrm{SSE}=Y^\mathsf{T}(I-P_X)Y$. The matrix $I-P_X$ is an orthogonal projection of rank $n-r$, so normal projection theory gives $U=\mathrm{SSE}/\sigma^2\sim\chi^2_{n-r}$. The contrast estimate depends on $P_XY$, whereas the residual sum of squares depends on $(I-P_X)Y$. These two projections are orthogonal; for a Gaussian vector their projected components are independent. Hence $Z$ and $U$ are independent. Since $$ T=\frac{Z}{\sqrt{U/(n-r)}}, $$ the defining representation of the Student t distribution gives $T\sim t_{n-r}$. $\square$

This establishes the reference distribution, not a choice among possible alternatives. Once an alternative is specified, the appropriate tail probability or critical region is determined from this same $t_{n-r}$ distribution.

Worked Calculations

Worked Example: Testing the Sum of Two Coefficients

Suppose $X$ has three orthogonal columns $h_0,h_1,h_2$, each of squared length 8, with $h_0$ the all-ones vector. The observed response is $Y=2h_0+h_1-\tfrac12h_2+2h_3$, where $h_3$ is orthogonal to all three design columns and has squared length 8. Test $H_0:\beta_1+\beta_2=0$, using coefficient order $(\beta_0,\beta_1,\beta_2)$.

Orthogonality gives $X^\mathsf{T}X=8I_3$, so $(X^\mathsf{T}X)^{-1}=\tfrac18 I_3$. The projection of $Y$ onto the design columns is $2h_0+h_1-\tfrac12h_2$, so $\hat\beta=(2,1,-\tfrac12)^\mathsf{T}$. For $a=(0,1,1)^\mathsf{T}$, the estimated contrast is $a^\mathsf{T}\hat\beta=1-\tfrac12=\tfrac12$.

The residual is $2h_3$, so $\mathrm{SSE}=4h_3^\mathsf{T}h_3=32$. Here $n=8$, $r=3$, and $\mathrm{MSE}=32/(8-3)=6.4$. The variance multiplier is $a^\mathsf{T}(X^\mathsf{T}X)^{-1}a=(1+1)/8=1/4$. Therefore the standard error is $\sqrt{6.4(1/4)}=\sqrt{1.6}$, and the observed statistic is $$ t_{\mathrm{obs}}=\frac{1/2-0}{\sqrt{1.6}}\approx 0.395, \qquad \text{with }5\text{ degrees of freedom}. $$

Worked Example: A Contrast Combining Intercept and Slope

In a simple linear regression with an intercept, suppose the predictor values are $-2,-1,0,1,2$, and the observed responses are $3,0.5,5,1.5,5$. Test the contrast $H_0:\beta_0+2\beta_1=3$.

The response vector can be written as $Y=3\mathbf{1}+0.5x+e$, where $e=(1,-2,2,-2,1)^\mathsf{T}$. Direct calculation gives $\mathbf{1}^\mathsf{T}e=0$ and $x^\mathsf{T}e=0$. Thus the least-squares estimates are $\hat\beta_0=3$ and $\hat\beta_1=0.5$. The contrast estimate is $3+2(0.5)=4$.

Here $$ X^\mathsf{T}X= \begin{pmatrix}5&0\\0&10\end{pmatrix}, \qquad (X^\mathsf{T}X)^{-1}= \begin{pmatrix}1/5&0\\0&1/10\end{pmatrix}. $$ With $a=(1,2)^\mathsf{T}$, the variance multiplier is $a^\mathsf{T}(X^\mathsf{T}X)^{-1}a=1/5+4/10=3/5$. The residual sum of squares is $1^2+(-2)^2+2^2+(-2)^2+1^2=14$, so $\mathrm{MSE}=14/(5-2)=14/3$. The standard error is therefore $\sqrt{(14/3)(3/5)}=\sqrt{14/5}\approx1.673$, and $$ t_{\mathrm{obs}}=\frac{4-3}{\sqrt{14/5}}\approx0.598, \qquad \text{with }3\text{ degrees of freedom}. $$

Worked Example: Including Covariance in the Standard Error

Consider a regression with an intercept and predictor values $0,1,2,3$. The responses are $2,2,4,8$. Test $H_0:\beta_0+\beta_1=2$.

The predictor has mean $1.5$. The response mean is $4$, and the slope estimate is $\sum_i(x_i-\bar x)(y_i-\bar y)/\sum_i(x_i-\bar x)^2=10/5=2$. Thus $\hat\beta_1=2$ and $\hat\beta_0=4-2(1.5)=1$. The contrast estimate is $1+2=3$.

The design cross-product matrix and its inverse are $$ X^\mathsf{T}X= \begin{pmatrix}4&6\\6&14\end{pmatrix}, \qquad (X^\mathsf{T}X)^{-1} =\frac1{20}\begin{pmatrix}14&-6\\-6&4\end{pmatrix}. $$ For $a=(1,1)^\mathsf{T}$, the variance multiplier is $14/20-6/20-6/20+4/20=6/20=0.3$. The two off-diagonal entries are covariance contributions and cannot be omitted. Fitted responses are $1,3,5,7$, giving residuals $1,-1,-1,1$ and $\mathrm{SSE}=4$. Thus $\mathrm{MSE}=4/(4-2)=2$, the standard error is $\sqrt{2(0.3)}=\sqrt{0.6}\approx0.775$, and $$ t_{\mathrm{obs}}=\frac{3-2}{\sqrt{0.6}}\approx1.291, \qquad \text{with }2\text{ degrees of freedom}. $$

Common Pitfall and Practical Interpretation

Wrong reasoning: “The contrast is the sum of two coefficient estimates, so its standard error is the sum of their standard errors.” Standard errors are not additive in that way. A contrast is a random variable whose variance depends on the covariance matrix of all coefficient estimates involved.

Correction: First calculate the variance multiplier $a^\mathsf{T}(X^\mathsf{T}X)^+a$, retaining its cross-product terms. Then multiply by $\mathrm{MSE}$ and take the square root. The exact test also depends on using the residual degrees of freedom $n-r$, where $r=\operatorname{rank}(X)$, rather than automatically using $n-p$ in a rank-deficient design.

The contrast test is useful whenever a model-based question can be expressed as one linear combination of parameters. Its exactness depends on the stated Gaussian, constant-variance, independent-error model with fixed design. If those conditions do not hold, the displayed statistic need not have an exact Student t distribution, even if the calculation itself is unchanged.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following questions.

In a full-rank model, how do you calculate the estimated standard error of $a^\mathsf{T}\hat\beta$?
Why can the covariance between two coefficient estimates change the standard error of their sum?
Under the null hypothesis, what distribution does the studentized contrast have, and what determines its degrees of freedom?
For a rank-deficient design, why must a contrast be estimable before its value can be tested?
In the third worked example, what contribution would be lost if the off-diagonal entries of $(X^\mathsf{T}X)^{-1}$ were ignored?