Conditional Exactness Under Gaussian Spherical Errors

Why the Design Must Be Held Fixed

A regression or analysis-of-variance calculation uses a design: the rows of a matrix record which predictors or treatment conditions apply to each observation. Exact small-sample t and F calibration is not a consequence of the formulas alone. It follows from a probability model for the errors given that design. This distinction matters whether the design was fixed by the study plan or generated randomly.

The central assumptions are that the mean response lies in the fitted model space and that, conditional on the design, the error vector is multivariate normal with covariance matrix $\sigma^2 I$. The scalar covariance makes errors spherical: no direction in observation space has a different error variance. Gaussianity then turns orthogonal error components into independent components. Those facts produce exact finite-sample distributions, not merely large-sample approximations.

Definition: In a fixed-design Gaussian linear model, the observed design matrix $X$ is treated as fixed, $Y=X\beta+\varepsilon$, and $$ \varepsilon\mid X \sim N(0,\sigma^2 I_n), \qquad \sigma^2>0. $$ Here $Y$ and $\varepsilon$ are $n$-vectors, $\beta$ is a fixed parameter vector, and $I_n$ is the $n\times n$ identity matrix. Thus $E[Y\mid X]=X\beta$, and the errors are conditionally independent with common variance $\sigma^2$. For a random design, the same statements are assumptions about the conditional distribution given $X$.

For the results below, assume $X$ has $p$ linearly independent columns and $n>p$. Write $P_X=X(X'X)^{-1}X'$ for the orthogonal projection onto the column space of $X$, and $R_X=I_n-P_X$ for the residual projection. The least-squares estimator is $\hat\beta=(X'X)^{-1}X'Y$, and the residual sum of squares is $\mathrm{SSE}=Y'R_XY$. These definitions make the geometry explicit: fitted values lie in the model space, while residuals lie in its orthogonal complement.

Exact Studentization for a Contrast

Let $c$ be a nonzero $p$-vector and consider the estimable linear function $c'\beta$. Full column rank ensures every such function is estimable. Its least-squares estimate is $c'\hat\beta$, with conditional variance $\sigma^2 c'(X'X)^{-1}c$. The unknown variance is replaced by the residual mean square $\mathrm{SSE}/(n-p)$. The reason this replacement produces an exact t distribution is the independence of the contrast error and the residual vector.

Theorem: Under the fixed-design Gaussian assumptions above, let $\nu=n-p$, and suppose the null hypothesis $c'\beta=\theta_0$ is true. Then $$ T=\frac{c'\hat\beta-\theta_0} {\sqrt{\{\mathrm{SSE}/\nu\}\,c'(X'X)^{-1}c}} \sim t_\nu $$ conditionally on $X$. The result is finite-sample exact for every $\sigma^2>0$.

Proof. Put $a=X(X'X)^{-1}c$. Then $c'\hat\beta-c'\beta=a'\varepsilon$, and $$ a'a=c'(X'X)^{-1}c. $$ Consequently $U=(c'\hat\beta-c'\beta)/\{\sigma\sqrt{c'(X'X)^{-1}c}\}$ has a standard normal distribution. Also, since $R_XX=0$, $\mathrm{SSE}=\varepsilon'R_X\varepsilon$. The matrix $R_X$ is symmetric and idempotent with rank $\nu$, so orthogonal diagonalization of the Gaussian error vector gives $\mathrm{SSE}/\sigma^2\sim\chi^2_\nu$.

The numerator error is a linear function of $\varepsilon$, and the residual vector is $R_X\varepsilon$. Their covariance is $$ \operatorname{Cov}(a'\varepsilon,R_X\varepsilon\mid X)=\sigma^2a'R_X=0, $$ because $a$ lies in the column space of $X$. These are jointly Gaussian, so zero covariance implies independence. In particular, $U$ is independent of $\mathrm{SSE}/\sigma^2$. Under the null, $c'\beta=\theta_0$, and therefore $$ T=\frac{U}{\sqrt{(\mathrm{SSE}/\sigma^2)/\nu}}. $$ The defining ratio of an independent standard normal and the square root of an independent $\chi^2_\nu/\nu$ variable has the $t_\nu$ distribution. This proves the claim conditionally on $X$. $\square$

Worked Example: A Difference Between Two Group Means

Suppose six observations follow a two-group model with an intercept and a group indicator. The observed values are $2,3,4$ in group A and $5,6,7$ in group B. Treat the group assignments as fixed, and consider the contrast $\beta_B-\beta_A$, where $\beta_A$ and $\beta_B$ are the two group means.

The fitted group means are $$ \bar Y_A=\frac{2+3+4}{3}=3,\qquad \bar Y_B=\frac{5+6+7}{3}=6. $$ The within-group residuals in each group are $-1,0,1$, so each group contributes $1+0+1=2$ to the residual sum of squares. Thus $\mathrm{SSE}=2+2=4$, $p=2$, and $\nu=6-2=4$. The residual mean square is $4/4=1$.

The estimated difference is $6-3=3$. Its variance factor, before multiplying by the residual variance, is $1/3+1/3=2/3$. The studentized statistic for testing equal means is therefore $$ t_{\mathrm{obs}}=\frac{3-0}{\sqrt{1(2/3)}}=3\sqrt{\frac{3}{2}}\approx 3.674. $$ Under the null of equal group means, conditional on these group assignments and the Gaussian spherical-error assumptions, this statistic has exactly a $t_4$ distribution. The distributional claim does not rely on an approximation in $n$.

Exact F Calibration for Nested Models

The same geometric idea gives an exact F distribution when comparing nested mean spaces. Let $\mathcal{M}_0$ be a subspace contained in a full model space $\mathcal{M}_1$, with dimensions $p-r$ and $p$, respectively. Let $P_0$ and $P_1$ be the orthogonal projection matrices onto these spaces. Under the null hypothesis, the true mean vector $\mu=E[Y\mid X]$ belongs to $\mathcal{M}_0$. The extra fitted component is measured by $Y'(P_1-P_0)Y$; the full-model residual component is $Y'(I_n-P_1)Y$.

Theorem: Suppose the conditional Gaussian spherical-error assumptions hold, $\mathcal{M}_0\subset\mathcal{M}_1$, $\dim(\mathcal{M}_1)=p$, $\dim(\mathcal{M}_0)=p-r$ with $r>0$, and the null mean lies in $\mathcal{M}_0$. If $n>p$, then $$ F= \frac{Y'(P_1-P_0)Y/r}{Y'(I_n-P_1)Y/(n-p)} \sim F_{r,n-p} $$ conditionally on the design.

Proof. Because the spaces are nested, $P_1P_0=P_0P_1=P_0$. Hence $A=P_1-P_0$ is symmetric and idempotent, with rank $r$; $B=I_n-P_1$ is also symmetric and idempotent, with rank $n-p$. Moreover $AB=0$. Under the null, $A\mu=0$ and $B\mu=0$, so the two quadratic forms in the statistic are $\varepsilon'A\varepsilon$ and $\varepsilon'B\varepsilon$.

For a Gaussian vector with covariance $\sigma^2 I_n$, an orthogonal change of coordinates preserves independent $N(0,\sigma^2)$ coordinates. Diagonalizing each projection therefore shows that $\varepsilon'A\varepsilon/\sigma^2\sim\chi^2_r$ and $\varepsilon'B\varepsilon/\sigma^2\sim\chi^2_{n-p}$. The condition $AB=0$ means their projected Gaussian components are orthogonal; their covariance is zero, so the two quadratic forms are independent. Dividing each by its degrees of freedom and taking their ratio cancels $\sigma^2$, giving the stated $F_{r,n-p}$ distribution. $\square$

Worked Example: Comparing an Intercept Model With a Linear Trend

For $n=4$, fit a full model with an intercept and the fixed predictor $z=(-3,-1,1,3)'$, and compare it with the null model containing only an intercept. Let $Y=(1,2,3,5)'$. The predictor sums to zero, so it is orthogonal to the intercept vector.

The null-model fit is the overall mean $\bar Y=11/4=2.75$. Its residual sum of squares is $$ \mathrm{SSE}_0=(-1.75)^2+(-0.75)^2+(0.25)^2+(2.25)^2=8.75. $$ For the full model, the intercept estimate is also $2.75$, and the slope estimate is $$ \hat\beta_z=\frac{z'Y}{z'z} =\frac{-3(1)-1(2)+1(3)+3(5)}{9+1+1+9} =\frac{13}{20}=0.65. $$ The fitted values are $0.80,2.10,3.40,4.70$. The residuals are $0.20,-0.10,-0.40,0.30$, giving $$ \mathrm{SSE}_1=(0.20)^2+(-0.10)^2+(-0.40)^2+(0.30)^2=0.30. $$ Thus the reduction in residual sum of squares is $8.75-0.30=8.45$. Here $r=1$, $p=2$, and $n-p=2$, so the observed statistic is $$ F=\frac{8.45/1}{0.30/2}=\frac{8.45}{0.15}\approx 56.33. $$ If the true mean has no linear trend, the statistic has exactly an $F_{1,2}$ distribution conditional on this design. This example computes the statistic; the null distribution is a separate consequence of the error assumptions.

What Conditioning Does—and Does Not—Guarantee

When $X$ is fixed by design, conditioning is implicit: the probability statements concern repeated outcomes $Y$ under that same $X$. When $X$ is random, the conditional model must still hold. In particular, the errors must have the stated Gaussian distribution given $X$, and the required ranks must hold. Conditioning does not repair a model in which the errors depend on the design in an unaccounted-for way.

Worked Example: Random Group Assignment and Conditional Calibration

Imagine a study with six participants in which assignment is randomized to either three in group A and three in group B, or two in group A and four in group B. Suppose both possible assignments have positive probability, assignment is independent of the errors, the two-group mean model is correct, and the errors are conditionally independent $N(0,\sigma^2)$.

For either assignment, the group-mean model has rank $p=2$, and the residual degrees of freedom are $6-2=4$. Under the null of equal group means, the studentized difference has a $t_4$ distribution conditional on either assignment. The estimated standard error uses the appropriate group sizes: its variance factor is $1/3+1/3=2/3$ for the first allocation and $1/2+1/4=3/4$ for the second.

For any fixed $t_0$, the law of total probability gives $$ P(T\le t_0) =\sum_x P(T\le t_0\mid X=x)P(X=x) =\sum_x F_{t_4}(t_0)P(X=x) =F_{t_4}(t_0). $$ Thus the statistic is also $t_4$ unconditionally in this example. The equality follows because the same conditional reference distribution holds for every possible assignment, not merely because the design happened to be randomized.

Common Pitfalls and Why Exactness Matters

Common mistake: “The sums of squares come from orthogonal projections, so the t and F results are exact even without Gaussian errors.” Orthogonality gives zero covariance under spherical errors, but zero covariance does not generally imply independence. The proofs above use joint Gaussianity to obtain independence of orthogonal components. Without it, the same statistics may be useful approximations under additional conditions, but the exact t and F claims do not follow from projection geometry alone.

A second pitfall is to treat $\sigma^2 I_n$ as a cosmetic assumption. If errors have unequal variances or nonzero correlations, the projection components need not have the chi-square distributions used in the proofs. The classical exact calibration is valuable precisely because, under its assumptions, it accounts for finite-sample uncertainty without relying on a limiting approximation. It is also conditional: if the design was selected using outcomes or errors, the stated conditional error model must be justified anew.

Key takeaway: Exact classical t and F calibration comes from the conditional Gaussian model with covariance $\sigma^2 I_n$, correct mean-space assumptions, and the required ranks. Orthogonal projections create the relevant components; Gaussianity makes the orthogonal components independent.

Check Your Understanding

Use the assumptions and projection arguments above to answer the following.

What does it mean to say that errors are spherical conditional on the design?
In the contrast proof, why is the contrast estimation error independent of the residual sum of squares?
For a nested-model comparison, what are the numerator and denominator degrees of freedom in terms of the model dimensions?
Why does orthogonality by itself not establish the exact t and F distributions?
If a random design takes several possible values, what condition allows the conditional t calibration to imply the same unconditional calibration?