Worked Confidence Regions for Contrasts in a Small Model

From a Fitted Model to a Contrast Region

In a normal linear model, an inferential question often concerns a particular combination of coefficients rather than each coefficient separately. For example, a study might ask whether the difference between two fitted effects is positive, or whether two specified combinations of effects lie near their estimated values. A confidence region for such contrasts turns the model’s coefficient estimates and residual variation into an interval or a joint region on the scale of the question.

The calculations below use the fixed-design normal linear model $Y\sim N(X\beta,\sigma^2 I_n)$, where $X$ is a known $n\times p$ matrix of full column rank, $\beta\in\mathbb{R}^p$, and $\sigma^2>0$. Assume $n>p$, so that the residual degrees of freedom are positive. Write $\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y$, $SSE=(Y-X\hat\beta)^\mathsf{T}(Y-X\hat\beta)$, $\nu=n-p$, and $s^2=SSE/\nu$.

Definition: A linear contrast is a specified linear function of the model coefficients, such as $c^\mathsf{T}\beta$, where $c\in\mathbb{R}^p$ is fixed. Under the full-column-rank assumption on $X$, every such function is estimable. For $c\ne 0$, its estimate is $c^\mathsf{T}\hat\beta$, and its variance is $\sigma^2 c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$. The estimated variance replaces $\sigma^2$ with $s^2$.

Exact Confidence Interval for One Contrast

The variance factor $c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$ depends on the contrast and the design, not on the observed responses. After estimating $\sigma^2$, the standard error is $s\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}$. The appropriate critical value comes from a Student $t$ distribution with $\nu=n-p$ degrees of freedom.

Theorem: Under the normal linear model above, for any fixed nonzero $c\in\mathbb{R}^p$, an exact $100(1-\alpha)\%$ confidence interval for $c^\mathsf{T}\beta$, where $0<\alpha<1$, is $$ c^\mathsf{T}\hat\beta \ \pm\ t_{\nu,\,1-\alpha/2}\, s\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}. $$ Here $t_{\nu,\,1-\alpha/2}$ is the $1-\alpha/2$ quantile of the $t$ distribution with $\nu$ degrees of freedom.

Proof. The least-squares estimator is normal with mean $\beta$ and covariance matrix $\sigma^2(X^\mathsf{T}X)^{-1}$. Thus \[ \frac{c^\mathsf{T}\hat\beta-c^\mathsf{T}\beta} {\sigma\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}} \sim N(0,1). \] The residual sum of squares satisfies $SSE/\sigma^2\sim\chi^2_\nu$ and is independent of $\hat\beta$. To see the independence, $X\hat\beta$ and the residual vector are orthogonal projections of the jointly normal vector $Y$ onto orthogonal subspaces; their covariance is zero, and jointly normal vectors with zero covariance are independent. Therefore replacing $\sigma$ in the standardized contrast by $s$ gives a $t_\nu$ statistic. The probability that this statistic lies between $-t_{\nu,1-\alpha/2}$ and $t_{\nu,1-\alpha/2}$ is $1-\alpha$. Rearranging that event for $c^\mathsf{T}\beta$ gives the stated interval. $\square$

Worked Example: A Scalar Contrast from a Two-Parameter Model

Suppose $n=5$, $p=2$, $X^\mathsf{T}X$ has inverse \[ (X^\mathsf{T}X)^{-1} = \begin{pmatrix} 0.6&-0.2\\ -0.2&0.1 \end{pmatrix}, \] the fitted coefficient vector is $\hat\beta=(6,4)^\mathsf{T}$, and $SSE=10$. Find a 95% confidence interval for $c^\mathsf{T}\beta$ when $c=(1,2)^\mathsf{T}$.

There are $\nu=n-p=5-2=3$ residual degrees of freedom, so $s^2=10/3$. The estimated contrast is $c^\mathsf{T}\hat\beta=1(6)+2(4)=14$.

Calculate the variance factor using the matrix-vector product: \[ \begin{pmatrix} 0.6&-0.2\\ -0.2&0.1 \end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} = \begin{pmatrix}0.2\\0\end{pmatrix}, \qquad c^\mathsf{T}(X^\mathsf{T}X)^{-1}c =(1,2)\begin{pmatrix}0.2\\0\end{pmatrix} =0.2. \] Thus the estimated variance is $s^2(0.2)=(10/3)(0.2)=2/3$, and the standard error is $\sqrt{2/3}\approx0.8165$. Since $t_{3,0.975}\approx3.1824$, the margin is $3.1824\sqrt{2/3}\approx2.598$. The interval is \[ 14\pm2.598=(11.402,\ 16.598), \] to three decimal places.

Joint Region for a Specified Vector of Contrasts

Several contrasts can be treated together. Let $C$ be a fixed $q\times p$ matrix of rank $q$, with $1\le q\le p$. The target is the vector $C\beta$, and its estimate is $C\hat\beta$. The covariance matrix apart from the unknown factor $\sigma^2$ is $G=C(X^\mathsf{T}X)^{-1}C^\mathsf{T}$. The rank assumptions make $G$ positive definite, so its inverse defines the shape of a $q$-dimensional ellipsoid.

Theorem: Under the normal linear model above, a joint $100(1-\alpha)\%$ confidence region for $C\beta$ is $$ \left\{\theta\in\mathbb{R}^q: (\theta-C\hat\beta)^\mathsf{T}G^{-1}(\theta-C\hat\beta) \le q\,s^2 F_{q,\nu;\,1-\alpha} \right\}, \qquad G=C(X^\mathsf{T}X)^{-1}C^\mathsf{T}, $$ where $F_{q,\nu;\,1-\alpha}$ is the $1-\alpha$ quantile of the $F$ distribution with $q$ and $\nu$ degrees of freedom.

Proof. The vector $C\hat\beta-C\beta$ is normal with mean zero and covariance $\sigma^2G$. Consequently, \[ Z=\sigma^{-1}G^{-1/2}(C\hat\beta-C\beta) \sim N_q(0,I_q), \] so $Z^\mathsf{T}Z\sim\chi^2_q$. As in the scalar result, $SSE/\sigma^2\sim\chi^2_\nu$ and is independent of $\hat\beta$, hence independent of $Z$. It follows from the definition of the $F$ distribution that \[ \frac{Z^\mathsf{T}Z/q}{(SSE/\sigma^2)/\nu} \sim F_{q,\nu}. \] Since $s^2=SSE/\nu$, the statistic on the left is \[ \frac{(C\hat\beta-C\beta)^\mathsf{T}G^{-1}(C\hat\beta-C\beta)} {q\,s^2}. \] It is at most $F_{q,\nu;\,1-\alpha}$ with probability $1-\alpha$. Rearranging this inequality gives precisely the stated region for $\theta=C\beta$. $\square$

Worked Example: A Joint Region for Two Coefficients

Let $n=8$, $p=2$, and suppose $(X^\mathsf{T}X)^{-1}=0.25I_2$, $\hat\beta=(2,1)^\mathsf{T}$, and $SSE=12$. Find the 95% joint region for $C\beta=\beta$, so $C=I_2$.

Here $\nu=8-2=6$, $s^2=12/6=2$, $q=2$, and $G=0.25I_2$, with $G^{-1}=4I_2$. Let $\theta=(\theta_1,\theta_2)^\mathsf{T}$. Using $F_{2,6;0.95}\approx5.143$, the region is \[ 4\{(\theta_1-2)^2+(\theta_2-1)^2\} \le 2(2)(5.143). \] Dividing both sides by four gives \[ (\theta_1-2)^2+(\theta_2-1)^2\le5.143. \] Thus the region is a disk centered at $(2,1)$ with radius $\sqrt{F_{2,6;0.95}}\approx2.268$.

For an exact boundary check, let $F_*=F_{2,6;0.95}$ and take $\theta=(2+\sqrt{F_*},1)^\mathsf{T}$. The left side of the original inequality is $4F_*$, and the right side is $4F_*$, so this point is exactly on the boundary. Using the rounded critical value, the point is approximately $(4.268,1)$; the equality is exact for the unrounded quantile and square root.

Evaluating a Contrast in a Three-Group Model

A useful variance calculation in a group-means model makes clear why each coefficient in a contrast contributes its squared weight divided by its group size. Suppose independent observations in group $i$ have mean $\mu_i$ and common variance $\sigma^2$, for $i=1,2,3$. With group sizes $n_i$, the model estimates are the sample means, and \[ \operatorname{Var}(\hat\mu_i)=\frac{\sigma^2}{n_i}. \] Because groups are independent, the variance of $\sum_i a_i\hat\mu_i$ is $\sigma^2\sum_i a_i^2/n_i$. The pooled residual variance estimates the common $\sigma^2$, with residual degrees of freedom equal to the total sample size minus three fitted means.

Worked Example: Comparing One Group with the Average of Two Others

Three independent groups have sizes $n_1=2,n_2=3,n_3=2$, observed means $\hat\mu_1=10,\hat\mu_2=13,\hat\mu_3=12$, and pooled $SSE=8$. Find a 95% confidence interval for $\mu_1-(\mu_2+\mu_3)/2$.

The contrast weights are $a=(1,-1/2,-1/2)$. Its estimate is $10-(13+12)/2=-2.5$. The total sample size is 7 and three group means are fitted, so $\nu=7-3=4$ and $s^2=8/4=2$.

The variance factor is \[ \frac{1^2}{2}+\frac{(-1/2)^2}{3}+\frac{(-1/2)^2}{2} =\frac12+\frac1{12}+\frac18 =\frac{17}{24}. \] Therefore the estimated variance is $2(17/24)=17/12$, and the standard error is $\sqrt{17/12}\approx1.1902$. With $t_{4,0.975}\approx2.7764$, the margin is about $2.7764(1.1902)=3.305$. The interval is \[ -2.5\pm3.305=(-5.805,\ 0.805), \] to three decimal places. The interval includes zero, so these data do not exclude equality of the first group mean and the average of the other two at the 5% two-sided level.

Common Mistakes and Interpretation

The scalar interval and joint region use the same fitted model but answer different questions. The scalar interval concerns one specified contrast. The ellipsoid concerns the vector of $q$ contrasts in the chosen matrix $C$, with joint coverage for that vector. It is not a region for every possible contrast, and it is not obtained by treating several separate 95% intervals as though they automatically had 95% joint coverage.

Using the wrong degrees of freedom. The residual variance estimate uses $\nu=n-p$, not $n$. If $n=p$, then $\nu=0$, and these estimated-variance $t$ and $F$ procedures are not defined.
Dropping off-diagonal terms. For a scalar contrast, calculate the full quadratic form $c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$. Correlations between coefficient estimates affect its value.
Using separate intervals for joint coverage. Individual intervals each have their stated marginal coverage; that fact alone does not establish the same coverage for all contrasts simultaneously.
Mixing up variance and standard error. First calculate $s^2c^\mathsf{T}(X^\mathsf{T}X)^{-1}c$; take its square root only when forming the interval margin.

Key takeaway: For a scalar contrast, combine its estimated value with a $t$ critical value and its design-based standard error. For a full-rank vector of specified contrasts, use the corresponding covariance-shape matrix and an $F$ critical value to obtain an exact joint ellipsoid.

Check Your Understanding

Use the assumptions and procedures developed above to answer these questions.

Why must $n>p$ for the estimated-variance confidence procedures in this tutorial?
For a scalar contrast $c^\mathsf{T}\beta$, which matrix expression gives the variance factor before multiplication by $\sigma^2$?
In the joint-region theorem, what determines the dimension of the $F$ distribution’s numerator degrees of freedom?
For the three-group example, recalculate the variance factor if the target is $\mu_1-\mu_2$, keeping the same group sizes.
Explain why the exact boundary point in the two-coefficient example uses the unrounded $F$ quantile rather than a rounded decimal.