Exact Intervals for Scalar Linear Contrasts

Why Construct an Interval for a Contrast?

A coefficient estimate gives one fitted value, but uncertainty matters when interpreting it. Often the scientific question concerns a particular linear combination of coefficients rather than a single coefficient: a slope, a difference between two coefficients, or another specified combination. An exact confidence interval for that quantity uses its estimated variance and the residual degrees of freedom.

The word exact has a specific meaning here. Under the normal linear model assumptions stated below, the interval has its advertised coverage probability for every sample size, not merely approximately for large samples. The interval is for one scalar contrast; it does not by itself give simultaneous coverage for several contrasts.

Model, Estimability, and Contrast Variance

Let $Y$ be an $n$-dimensional response vector, $X$ a fixed $n\times p$ design matrix of full column rank, and $\beta$ a $p$-dimensional coefficient vector. Assume

Y=X\beta+\varepsilon,\qquad \varepsilon\sim N_n(0,\sigma^2 I_n),\qquad \sigma^2>0,\qquad n>p.

The full-rank assumption makes every coefficient combination $c^T\beta$, for a specified vector $c\in\mathbb{R}^p$, estimable. In a rank-deficient design, only certain combinations are estimable; the formulas below apply to an estimable contrast when its least-squares estimate and variance are defined.

Definition: A scalar linear contrast is a specified quantity $c^T\beta$, where $c$ is a fixed, nonzero $p$-vector. Its least-squares estimate is $c^T\hat{\beta}$, with $\hat{\beta}=(X^TX)^{-1}X^TY$. Write $$ q=c^T(X^TX)^{-1}c. $$ Then $q>0$, and the estimated standard error of $c^T\hat{\beta}$ is $s\sqrt{q}$, where $s^2=\mathrm{SSE}/(n-p)$ and $\mathrm{SSE}=(Y-X\hat{\beta})^T(Y-X\hat{\beta})$.

The contrast variance depends on the entire coefficient covariance structure. In particular,

\operatorname{Var}(c^T\hat{\beta})=\sigma^2c^T(X^TX)^{-1}c=\sigma^2q.

Replacing $\sigma^2$ by $s^2$ gives the estimated variance $s^2q$. The square root is the estimated standard error used to measure uncertainty on the same scale as the contrast.

Worked Example: An Exact Interval for a Slope

Consider a simple regression with an intercept and a slope, six fixed predictor values $x=(-5,-3,-1,1,3,5)$, and responses $y_i=10+2x_i+e_i$, where $e=(1,-1,-1,1,0,0)$. Find a 95% interval for the slope.

The contrast is the slope, so $c=(0,1)^T$. First check the least-squares fit. The residuals sum to zero, and their inner product with the predictor values is $-5+3+1+1=0$. Thus they are orthogonal to both design columns, so $\hat{\beta}=(10,2)^T$. Also, $\sum_i x_i=0$, $\sum_i x_i^2=70$, and $\mathrm{SSE}=1+1+1+1=4$.

Here $n=6$, $p=2$, and the residual degrees of freedom are $4$. Therefore $s^2=4/4=1$. Since $X^TX$ is diagonal with entries $6$ and $70$, $q=1/70$ for the slope contrast. Its estimated standard error is $1/\sqrt{70}$. Using $t_{0.975,4}\approx2.776$, the interval is

2\pm 2.776\sqrt{\frac{1}{70}} \;=\;2\pm0.332 \;=\;(1.668,\;2.332).

Under the stated normal model, this interval has exactly 95% coverage for the slope. The numerical endpoints are rounded; the unrounded interval is $2\pm t_{0.975,4}/\sqrt{70}$.

The Exact t Pivot

The interval construction follows from a pivot: a statistic whose distribution does not depend on the unknown coefficient vector or variance. The numerator comes from the contrast estimate, while the denominator estimates its standard deviation.

Theorem: Under the fixed-design normal linear model above, for any nonzero $c$, $$ T=\frac{c^T\hat{\beta}-c^T\beta}{s\sqrt{c^T(X^TX)^{-1}c}}\sim t_{n-p}. $$

Proof. Let $P_X=X(X^TX)^{-1}X^T$, the orthogonal projection onto the column space of $X$. The least-squares estimator satisfies $\hat{\beta}-\beta=(X^TX)^{-1}X^T\varepsilon$, so $c^T\hat{\beta}-c^T\beta$ is normal with mean zero and variance $\sigma^2q$. Consequently,

Z=\frac{c^T\hat{\beta}-c^T\beta}{\sigma\sqrt q}\sim N(0,1).

The residual vector is $(I_n-P_X)Y=(I_n-P_X)\varepsilon$, and $\mathrm{SSE}=\varepsilon^T(I_n-P_X)\varepsilon$. The matrix $I_n-P_X$ is an orthogonal projection of rank $n-p$. Because $\varepsilon/\sigma\sim N_n(0,I_n)$, an orthogonal change of coordinates diagonalizes this projection, with $n-p$ diagonal entries equal to one and the others zero. Thus

U=\frac{\mathrm{SSE}}{\sigma^2}\sim\chi^2_{n-p}.

It remains to establish independence. The random vectors $P_X\varepsilon$ and $(I_n-P_X)\varepsilon$ are jointly normal and have cross-covariance $\sigma^2P_X(I_n-P_X)=0$. Joint normality and zero cross-covariance imply independence. The numerator of $Z$ is a function of $P_X\varepsilon$, while $U$ is a function of $(I_n-P_X)\varepsilon$, so $Z$ and $U$ are independent. Since $s^2=\sigma^2U/(n-p)$, it follows that

T=\frac{Z}{\sqrt{U/(n-p)}}\sim t_{n-p}.

This proves the pivot result. $\square$

Inverting the Pivot to Obtain an Interval

For a confidence level $1-\alpha$, where $0<\alpha<1$, let $t_{1-\alpha/2,n-p}$ denote the $1-\alpha/2$ quantile of the $t$ distribution with $n-p$ degrees of freedom. Its central probability is $1-\alpha$:

P\!\left(-t_{1-\alpha/2,n-p}\leq T\leq t_{1-\alpha/2,n-p}\right)=1-\alpha.

Theorem: An exact two-sided $100(1-\alpha)\%$ confidence interval for $c^T\beta$ is $$ c^T\hat{\beta}\ \pm\ t_{1-\alpha/2,n-p}\,s\sqrt{c^T(X^TX)^{-1}c}. $$

Derivation and coverage proof. Substitute the pivot expression into the event bounded by the two critical values. Since $s\sqrt q>0$, multiplying all sides by this positive quantity and rearranging gives the equivalent event

c^T\hat{\beta}-t_{1-\alpha/2,n-p}s\sqrt q \leq c^T\beta\leq c^T\hat{\beta}+t_{1-\alpha/2,n-p}s\sqrt q.

The pivot theorem assigns probability $1-\alpha$ to the original event. Equivalent events have the same probability, so the displayed interval contains the fixed, unknown $c^T\beta$ with probability exactly $1-\alpha$. The probability statement is about repeated responses under the model, with the design fixed; it does not mean that the fixed parameter has a probability of $1-\alpha$ of lying in a particular realized interval. $\square$

For a two-sided 95% interval, use $t_{0.975,n-p}$, not automatically the standard normal critical value $1.96$. The critical value depends on the residual degrees of freedom; it approaches the normal critical value as those degrees of freedom grow.

Worked Example: A Sum of Correlated Coefficients

Let $n=5$, with design columns $x_1=(1,1,0,0,0)^T$ and $x_2=(1,0,1,0,0)^T$. Take $Y=X(3,1)^T+e$, where $e=(0,0,0,\sqrt{6},0)^T$. Find a 95% interval for $\beta_1+\beta_2$.

The design has $X^TX=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, and its inverse is $\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. The residual vector is orthogonal to both columns, so $\hat{\beta}=(3,1)^T$ and $\mathrm{SSE}=6$. Thus $p=2$, the residual degrees of freedom are $5-2=3$, and $s^2=6/3=2$.

For the sum, $c=(1,1)^T$, and the estimate is $c^T\hat{\beta}=4$. The contrast variance factor is

q=c^T(X^TX)^{-1}c =\frac{2}{3}+\frac{2}{3}+2\left(-\frac{1}{3}\right) =\frac{2}{3}.

The off-diagonal terms matter: omitting the covariance contribution would give the wrong variance. The estimated standard error is $s\sqrt q=\sqrt{4/3}=2/\sqrt{3}\approx1.155$. Since $t_{0.975,3}\approx3.182$, the interval is

4\pm3.182\left(\frac{2}{\sqrt3}\right) \approx4\pm3.674 =(0.326,\;7.674).

The interval is wide because only three residual degrees of freedom are available. Its wide span does not alter the exact coverage property.

Worked Example: A Contrast in an Orthonormal Design

Suppose $n=4$, $p=2$, and the two columns of $X$ are $x_1=(1,1,1,1)^T/2$ and $x_2=(1,-1,1,-1)^T/2$. Let $Y=X(1.5,0.5)^T+e$, where $e=\sqrt6(1,-1,-1,1)^T/2$. Construct a 95% interval for $\beta_1-\beta_2$.

The columns have unit length and are orthogonal, so $X^TX=I_2$. The residual vector is orthogonal to both columns and has squared length $6$. Hence $\hat{\beta}=(1.5,0.5)^T$, $\mathrm{SSE}=6$, $n-p=2$, and $s^2=6/2=3$.

Here $c=(1,-1)^T$, so the estimated contrast is $1.5-0.5=1$, and $q=c^Tc=2$. Its standard error is $\sqrt{3\cdot2}=\sqrt6\approx2.449$. With $t_{0.975,2}\approx4.303$, the interval is

1\pm4.303\sqrt6\approx1\pm10.542=(-9.542,\;11.542).

This interval includes zero and is quite imprecise. Exact calibration does not guarantee a narrow interval; the width reflects both the estimated noise and the small residual degrees of freedom.

Common Mistakes and Interpretation

Using a normal critical value by habit. The exact pivot uses $t_{n-p}$ because $\sigma^2$ is estimated. Replacing its critical value by a normal quantile is generally an approximation, not the exact interval.
Dropping the contrast variance factor. The standard error is $s\sqrt{c^T(X^TX)^{-1}c}$, not just $s$. The contrast determines how coefficient uncertainty combines.
Ignoring covariance terms. For a sum or difference of coefficients, the off-diagonal entries of $(X^TX)^{-1}$ contribute to $q$, as in the second worked example.
Using the wrong degrees of freedom. The denominator degrees of freedom are $n-p$, the residual degrees of freedom in this full-rank model, not $n$ and not the number of coefficients in the contrast.
Confusing separate intervals with joint coverage. Each scalar interval has its stated marginal coverage. Constructing several such intervals does not automatically give the same coverage for the event that they all contain their targets.

The practical value of a contrast interval is that it answers a question on its natural scale. Its center is the fitted contrast, and its half-width combines the appropriate standard error with a finite-sample critical value. Always state which contrast is being estimated, report the interval in context, and distinguish its pointwise coverage from any claim about several quantities at once.

Check Your Understanding

Use the model assumptions and formulas in this tutorial to answer the following.

For a full-rank model with $n=18$ observations and $p=4$ coefficients, what degrees of freedom determine the exact $t$ critical value?
For a contrast $c^T\beta$, explain why its estimated standard error depends on $c^T(X^TX)^{-1}c$, not just on $s$.
In the second worked example, what would go wrong if the off-diagonal entries of $(X^TX)^{-1}$ were omitted when calculating the variance of the sum?
Why are the standardized coefficient error and the residual sum of squares independent under the stated model?
Does a 95% interval for each of two contrasts necessarily provide 95% probability that both intervals cover their targets? Explain.