Deriving the Exact Contrast Ellipsoid

Why Derive the Ellipsoid?

A joint confidence region for several regression contrasts must account for their covariance as well as their individual uncertainty. Under the normal linear model, that covariance has a precise form, and the residual variance estimate is independent of the estimated contrasts. Together, these facts yield an exact finite-sample calibration: an F distribution, not a large-sample approximation.

The derivation also identifies when the result applies. In particular, there must be residual degrees of freedom to estimate the error variance. The full-column-rank condition alone does not guarantee this: the sample size must exceed the number of regression coefficients.

Model and Contrast Setup

Definition: Let $Y\in\mathbb{R}^n$ satisfy the fixed-design normal linear model $$ Y=X\beta+\varepsilon,\qquad \varepsilon\sim N_n(0,\sigma^2 I_n), $$ where $X$ is a known $n\times p$ matrix of rank $p$, $n>p$, and $\sigma^2>0$. The least-squares estimator and residual variance estimate are $$ \hat{\beta}=(X'X)^{-1}X'Y,\qquad \mathrm{SSE}=(Y-X\hat{\beta})'(Y-X\hat{\beta}),\qquad s^2=\frac{\mathrm{SSE}}{n-p}. $$ Let $C$ be a known $q\times p$ matrix of rank $q$, with $1\le q\le p$. The vector of contrasts of interest is $\tau=C\beta$, estimated by $\hat{\tau}=C\hat{\beta}$. Define $$ A=C(X'X)^{-1}C'. $$

Because $X$ has full column rank, $X'X$ is positive definite. Since $C$ has full row rank, $A$ is also positive definite: for any nonzero $u\in\mathbb{R}^q$, $C'u\ne0$, so $u'Au=(C'u)'(X'X)^{-1}(C'u)>0$. Thus $A^{-1}$ exists and measures squared departures in the covariance-adjusted coordinates of the contrast vector.

Distribution and Independence

The estimator error is $\hat{\beta}-\beta=(X'X)^{-1}X'\varepsilon$. It follows directly from the normal model that $$ \hat{\tau}-\tau=C(X'X)^{-1}X'\varepsilon \sim N_q(0,\sigma^2 A). $$ Consequently, whitening this vector gives a sum of $q$ independent squared standard normal variables.

Theorem: Under the model and rank assumptions above, $$ \frac{(\hat{\tau}-\tau)'A^{-1}(\hat{\tau}-\tau)}{\sigma^2}\sim\chi^2_q, \qquad \frac{(n-p)s^2}{\sigma^2}\sim\chi^2_{n-p}, $$ and these two random variables are independent.

Proof. Let $P_X=X(X'X)^{-1}X'$ be the orthogonal projection onto the column space of $X$, and let $M_X=I_n-P_X$. The contrast estimation error is a linear function of $P_X\varepsilon$, because $$ \hat{\tau}-\tau=C(X'X)^{-1}X'P_X\varepsilon =C(X'X)^{-1}X'\varepsilon. $$ The residual vector is $Y-X\hat{\beta}=M_XY=M_X\varepsilon$, since $M_XX=0$. The two projected error vectors have covariance $$ \operatorname{Cov}(P_X\varepsilon,M_X\varepsilon) =\sigma^2P_XM_X=0. $$ They are jointly normal, as linear transformations of a normal vector. Zero cross-covariance therefore implies that $P_X\varepsilon$ and $M_X\varepsilon$ are independent. In particular, $\hat{\tau}-\tau$ and $\mathrm{SSE}=\varepsilon'M_X\varepsilon$ are independent.

The covariance of $\hat{\tau}-\tau$ is $\sigma^2A$, so $\sigma^{-1}A^{-1/2}(\hat{\tau}-\tau)$ has distribution $N_q(0,I_q)$. Its squared Euclidean length has a $\chi^2_q$ distribution. For the residual result, $M_X$ is symmetric and idempotent with rank $n-p$. An orthogonal change of coordinates therefore expresses $\varepsilon'M_X\varepsilon/\sigma^2$ as the sum of squares of $n-p$ independent standard normal variables. This is $\chi^2_{n-p}$, and the independence already shown applies to these functions of the independent projections. $\square$

From the Two Chi-Square Variables to the Ellipsoid

By definition, if independent variables $U\sim\chi^2_q$ and $V\sim\chi^2_\nu$, then $(U/q)/(V/\nu)\sim F_{q,\nu}$. Apply this with $\nu=n-p$, using $s^2/\sigma^2=V/(n-p)$. The result is the exact pivot

\frac{(\hat{\tau}-\tau)'A^{-1}(\hat{\tau}-\tau)}{q\,s^2} \sim F_{q,n-p}.

Write $F_{q,n-p;1-\alpha}$ for the $1-\alpha$ quantile of this F distribution. Since the pivot has a continuous distribution, its probability of being no greater than this quantile is exactly $1-\alpha$. Solving that inequality for the possible values of $\tau$ gives the confidence region.

Theorem: For $0<\alpha<1$, the exact $100(1-\alpha)\%$ confidence region for $\tau=C\beta$, conditional on the fixed design matrix $X$, is $$ \left\{\tau\in\mathbb{R}^q: (\hat{\tau}-\tau)'A^{-1}(\hat{\tau}-\tau) \le q\,s^2F_{q,n-p;1-\alpha}\right\}. $$ Its coverage probability is $1-\alpha$ under the stated normal-model assumptions.

The coverage claim follows because the true $\tau$ belongs to the displayed set precisely when the F pivot is at most its $1-\alpha$ quantile. Its distribution is exact for every fixed full-rank $X$ meeting the assumptions; no large-sample approximation is needed.

Worked Calculations

Worked Example: A Joint Region for Two Coefficients

Suppose $n=12$, $p=2$, and the design has orthogonal columns with $X'X=12I_2$. Take $C=I_2$, so $\tau=\beta$. Suppose the observed fit gives $\hat{\beta}=(2,1)'$ and $\mathrm{SSE}=20$.

There are $n-p=10$ residual degrees of freedom, and $s^2=20/10=2$. The contrast covariance factor is $A=C(X'X)^{-1}C'=I_2/12$, so $A^{-1}=12I_2$. At 95% confidence, $F_{2,10;0.95}\approx4.1028$. The region is therefore $$ \frac{12\{(\beta_1-2)^2+(\beta_2-1)^2\}}{2(2)} \le 4.1028. $$ Equivalently, $$ (\beta_1-2)^2+(\beta_2-1)^2 \le \frac{4(4.1028)}{12}\approx1.3676. $$ Thus the region is a disk centered at the fitted coefficient vector, with squared radius approximately $1.3676$. The reduction to a disk here follows from the equal, uncorrelated coefficient scales; it is not the general shape.

Worked Example: Correlated Linear Contrasts

Let $n=15$, $p=3$, $X'X=\operatorname{diag}(4,9,16)$, and let $$ C=\begin{pmatrix}1&0&1\\0&1&-1\end{pmatrix}. $$ The two targets are $\tau_1=\beta_1+\beta_3$ and $\tau_2=\beta_2-\beta_3$. Suppose $\hat{\beta}=(1,2,-1)'$ and $\mathrm{SSE}=24$. Then $\hat{\tau}=(0,3)'$, the residual degrees of freedom are $12$, and $s^2=24/12=2$.

First calculate the covariance factor: $$ A=C\begin{pmatrix}1/4&0&0\\0&1/9&0\\0&0&1/16\end{pmatrix}C' =\begin{pmatrix}5/16&-1/16\\-1/16&25/144\end{pmatrix}. $$ Its determinant is $(5/16)(25/144)-(1/16)^2=125/2304-9/2304=29/576>0$, and its inverse is $$ A^{-1}=\frac{1}{29}\begin{pmatrix}100&36\\36&180\end{pmatrix}. $$ Since $F_{2,12;0.95}\approx3.885$, the 95% region consists of those $\tau=(\tau_1,\tau_2)'$ satisfying $$ \frac{1}{29} \begin{pmatrix}-\tau_1&3-\tau_2\end{pmatrix} \begin{pmatrix}100&36\\36&180\end{pmatrix} \begin{pmatrix}-\tau_1\\3-\tau_2\end{pmatrix} \le 2(2)(3.885)=15.540. $$ The off-diagonal entries in the inverse matrix matter: the two contrast estimates are not uncorrelated, so separate univariate intervals would not reproduce this joint region.

Worked Example: Checking a Joint Hypothesis with the Same Pivot

Suppose $n=10$, $p=3$, $X'X=I_3$, and $C$ selects the first two coefficients. Thus $A=I_2$. Let the fitted contrast be $\hat{\tau}=(1,2)'$, the null value be $\tau_0=(0,0)'$, and let $\mathrm{SSE}=7$, giving $s^2=7/(10-3)=1$.

The observed pivot evaluated at the null is $$ \frac{(\hat{\tau}-\tau_0)'A^{-1}(\hat{\tau}-\tau_0)}{2s^2} =\frac{1^2+2^2}{2(1)}=2.5. $$ At the 5% level, the critical value is $F_{2,7;0.95}\approx4.737$. Since $2.5<4.737$, the null value lies within the 95% joint confidence region and is not rejected by the corresponding exact joint test. The region and test use the same statistic and cutoff; only the question asked of them differs.

Why the Conditions and Calibration Matter

The residual variance is estimated from the data, so the pivot has an F distribution rather than the chi-square distribution that would arise if $\sigma^2$ were known. The denominator degrees of freedom are $n-p$, because fitting $p$ linearly independent regression coefficients uses $p$ dimensions of the observation space. If $n=p$, then the residual projection has rank zero: $\mathrm{SSE}=0$ in this model and the formula for $s^2$ divides by zero. The exact ellipsoid derived here is consequently not defined.

A common mistake is to treat the estimated contrasts as independent merely because each one has a familiar standard error, or to replace $A$ by a diagonal matrix of marginal variances. The correction is to use the full covariance factor $A$ in the quadratic form. Another mistake is to use a chi-square cutoff while still estimating $\sigma^2$. The independence proof justifies the correct F calibration, including its numerator and denominator degrees of freedom.

Key takeaway: With full-rank $X$, $n>p$, a full-row-rank contrast matrix $C$, and normal errors of common variance, the whitened contrast error and residual variance estimate yield an exact $F_{q,n-p}$ pivot. Inverting that pivot gives the joint confidence ellipsoid.

Check Your Understanding

Use the assumptions and derivation above to answer these questions.

Why is $C(X'X)^{-1}C'$ positive definite when $C$ has full row rank?
Which two projected error components establish independence of the contrast estimator and residual sum of squares?
What are the numerator and denominator degrees of freedom in the exact pivot, and what determines each?
Why does the stated result require $n>p$, not just that $X$ have full column rank?
In the second worked example, what information would be lost by replacing $A$ with a diagonal matrix?