Cochran Decompositions Conditional on a Fixed Design

Why Conditioning on the Design Matters

In a linear model, fitted and residual sums of squares are determined by the design matrix through projections onto model spaces. If the design is fixed, Cochran’s theorem gives exact distributions for suitable orthogonal components of the error vector. If the design is random, those projection matrices are random too. The correct way to apply the theorem is to fix the realized design by conditioning on it, then check whether the error distribution still meets the theorem’s assumptions.

The central issue is not whether the design matrix was originally random. It is whether, after conditioning on that matrix, the errors are Gaussian with spherical covariance, and whether each sum of squares being called central is unaffected by the conditional mean.

Definition: A fixed-design Gaussian linear model has response vector $Y$, fixed design matrix $X$, parameter vector $\beta$, and error vector $\varepsilon$, with $$ Y=X\beta+\varepsilon,\qquad \varepsilon\sim N_n(0,\sigma^2 I_n),\quad \sigma^2>0. $$ Here $I_n$ is the $n\times n$ identity matrix. A projection matrix is a symmetric idempotent matrix $P$, so $P'=P$ and $P^2=P$; it projects onto its column space, and its rank is the dimension of that space.

Cochran’s Theorem for Orthogonal Projections

The projection form of Cochran’s theorem describes sums of squares of a spherical Gaussian vector. Orthogonality is a property of the projection matrices: two components are orthogonal when their product is zero. This makes their projected Gaussian vectors independent.

Theorem: Suppose $\varepsilon\sim N_n(0,\sigma^2 I_n)$, and $A_1,\ldots,A_m$ are symmetric idempotent matrices satisfying $A_iA_j=0$ for $i\ne j$. Then the quadratic forms $\varepsilon'A_i\varepsilon/\sigma^2$ are mutually independent, and $$ \frac{\varepsilon'A_i\varepsilon}{\sigma^2}\sim\chi^2_{r_i}, \qquad r_i=\operatorname{rank}(A_i). $$ If the projections sum to $I_n$, then $\sum_i r_i=n$. More generally, if they sum to a projection $M$, they partition the error sum of squares $\varepsilon'M\varepsilon$.

To prove the theorem, choose an orthonormal basis for the range of each $A_i$. The ranges are mutually orthogonal: if $u=A_i a$ and $v=A_j b$, then $u'v=a'A_iA_jb=0$. Extend these bases, if needed, to an orthonormal basis of $\mathbb R^n$. In the resulting coordinates, the transformed error vector still has independent $N(0,\sigma^2)$ entries, because an orthogonal transformation preserves its covariance $\sigma^2 I_n$ and its multivariate normal distribution.

In those coordinates, $\varepsilon'A_i\varepsilon$ is the sum of squares of the $r_i$ coordinates belonging to the range of $A_i$. Distinct matrices use disjoint coordinate sets, so the sums are independent. Dividing each by $\sigma^2$ gives a sum of $r_i$ independent squared standard normal variables, which has the $\chi^2_{r_i}$ distribution. If the projections sum to $I_n$, their mutually orthogonal ranges span $\mathbb R^n$, so their dimensions sum to $n$. $\square$

Applying the Theorem to Nested Model Spaces

Let $P_j$ be the orthogonal projection onto the $j$th model space in a nested sequence, with $P_0\le P_1\le\cdots\le P_k$. The notation $P_{j-1}\le P_j$ means that the range of $P_{j-1}$ is contained in the range of $P_j$. The increment $D_j=P_j-P_{j-1}$ projects onto the part added at step $j$, and $R=I_n-P_k$ projects onto the residual space.

D_j=P_j-P_{j-1},\qquad R=I_n-P_k.

These are mutually orthogonal projections. For example, using $P_{j-1}P_j=P_{j-1}$,

D_j^2=(P_j-P_{j-1})^2 =P_j-P_{j-1}-P_{j-1}+P_{j-1} =P_j-P_{j-1}=D_j.

They are symmetric because each $P_j$ is symmetric. Nestedness also gives $D_iD_j=0$ for $i\ne j$, and $D_jR=0$. Finally, $P_0+\sum_{j=1}^kD_j+R=I_n$. Thus the error vector’s quadratic forms in these projections satisfy Cochran’s theorem, with degrees of freedom equal to their ranks.

For sums of squares computed from $Y$, one additional check is essential. Write the conditional or fixed mean as $\mu=X\beta$. For a symmetric projection $A$,

Y'AY=(\mu+\varepsilon)'A(\mu+\varepsilon) =\mu'A\mu+2\mu'A\varepsilon+\varepsilon'A\varepsilon.

If $A\mu=0$, the first two terms vanish, so $Y'AY=\varepsilon'A\varepsilon$, which is central chi-square after division by $\sigma^2$. In particular, if the mean lies in the baseline space $\operatorname{range}(P_0)$, then every increment $D_j$ and the residual projection $R$ annihilate it. Their sums of squares are mutually independent and have central chi-square distributions. If a component does not annihilate the mean, its quadratic form is generally noncentral; orthogonality alone does not make it central.

Worked Example: A Fixed Design with a Baseline Mean

Let $u_1,u_2,u_3,u_4$ be the following orthonormal vectors in $\mathbb R^4$:

u_1=\tfrac12(1,1,1,1)',\quad u_2=\tfrac12(1,-1,1,-1)',\quad u_3=\tfrac12(1,1,-1,-1)',\quad u_4=\tfrac12(1,-1,-1,1)'.

Take $X=[u_1\ u_2]$, so the fitted model space is spanned by $u_1,u_2$. Use the baseline space spanned by $u_1$. The increment projection is $D=u_2u_2'$, and the residual projection is $R=u_3u_3'+u_4u_4'$. Their ranks are 1 and 2, respectively, and they are orthogonal.

Under the null model mean $\mu=5u_1$, both $D\mu=0$ and $R\mu=0$. Thus, conditional on this fixed design, Cochran’s theorem gives $Y'DY/\sigma^2\sim\chi^2_1$ and $Y'RY/\sigma^2\sim\chi^2_2$, independently.

For an observed vector $Y=5u_1+2u_2+u_3-u_4$, its coordinates in this basis are $5,2,1,-1$. Therefore the increment sum of squares is $Y'DY=2^2=4$, and the residual sum of squares is $Y'RY=1^2+(-1)^2=2$. These are observed values, not distributional claims; the central chi-square conclusions rely on the stated null mean and Gaussian spherical errors.

Random Designs: Apply the Result Conditionally

Now let $X$ be random, and suppose the projection matrices are measurable functions of $X$. A standard sufficient assumption is that, conditional on $X$, the errors satisfy $\varepsilon\mid X\sim N_n(0,\sigma^2I_n)$, with $\sigma^2$ fixed. The conditional mean is $X\beta$, and the projections used for central sums of squares must annihilate that mean. Under these assumptions, for each design realization where the required ranks and projection identities hold, the fixed-design theorem applies to the conditional distribution.

Conditional conclusion: If $A_1(X),\ldots,A_m(X)$ are measurable, mutually orthogonal symmetric projections and $\varepsilon\mid X\sim N_n(0,\sigma^2I_n)$, then, conditional on $X$, the quadratic forms $\varepsilon'A_i(X)\varepsilon/\sigma^2$ are independent with distributions $\chi^2_{\operatorname{rank}(A_i(X))}$. For response-based forms $Y'A_i(X)Y$, this central conclusion additionally requires $A_i(X)X\beta=0$ almost surely.

The conditional argument is direct. Fix a design realization $X=x$ for which the assumptions hold. The matrices $A_i(x)$ are then fixed orthogonal projections, and the conditional error distribution is $N_n(0,\sigma^2I_n)$. The theorem therefore gives the stated conditional joint distribution at that $x$. Since this holds for almost every $x$, it gives the conditional result given $X$. This argument does not require the projection matrices to be independent of the errors unconditionally; it requires the specified conditional error distribution.

Independence of $X$ and $\varepsilon$, together with $\varepsilon\sim N_n(0,\sigma^2I_n)$, is a familiar sufficient condition: it implies the required conditional distribution. Conditional assumptions can be stated more generally, but merely saying that errors are Gaussian before conditioning is not enough if their conditional distribution given $X$ changes.

Worked Example: Random Covariates with Fixed Rank

Suppose $n=5$, the model includes an intercept and one random covariate $x=(x_1,\ldots,x_5)'$, and $X=[\mathbf 1\ x]$. Assume the $x_i$ are independent continuous random variables and are independent of errors $\varepsilon\sim N_5(0,\sigma^2I_5)$. With probability one, the covariates are not all equal, so $\operatorname{rank}(X)=2$ almost surely. The residual projection is $R(X)=I_5-P_X$, where $P_X$ projects onto the column space of $X$; its rank is 3.

For every realized full-rank design, $R(X)X\beta=0$. Conditional on that design, $$ \frac{Y'R(X)Y}{\sigma^2} =\frac{\varepsilon'R(X)\varepsilon}{\sigma^2} \sim\chi^2_3. $$ Thus the conditional residual sum of squares has the same central distribution for almost every design. Because this conditional distribution does not depend on $X$, the residual sum of squares divided by $\sigma^2$ also has a $\chi^2_3$ distribution unconditionally. If the rank varied across designs, the conditional degrees of freedom could vary, and the unconditional law would generally be a mixture instead.

Worked Example: Why a Conditional Error Assumption Is Needed

Suppose a random design has two possible values, $X=a$ and $X=b$, each with probability $1/2$. Let $u$ be a fixed unit vector orthogonal to both design spaces, and consider the residual component $A=uu'$. Let the errors be conditionally Gaussian with mean zero, but have variance $\sigma^2$ given $X=a$ and $4\sigma^2$ given $X=b$. Then $$ \frac{\varepsilon'A\varepsilon}{\sigma^2}\,\bigg|\,X=a\sim\chi^2_1,\qquad \frac{\varepsilon'A\varepsilon}{\sigma^2}\,\bigg|\,X=b\sim4\chi^2_1. $$

The component is central chi-square at design $a$, but at design $b$ the same scaling does not produce a chi-square distribution. Conditional Gaussianity by itself is insufficient: the spherical covariance must have the stated scale, or the quadratic form must be standardized using the appropriate conditional variance. Unconditionally, the displayed quantity is a mixture of the two conditional laws, not a $\chi^2_1$ variable.

Common Mistake: Orthogonality Does Not Guarantee Centrality

A common error is to see that two projections are orthogonal and conclude that their response-based sums of squares are independent central chi-square variables. Orthogonality supports independence under conditional Gaussianity, but centrality also depends on the mean. For a projection $A$, the relevant check is $A\mu=0$, not merely $A B=0$ for another projection $B$. If $A\mu\ne0$, the projected Gaussian vector has a nonzero mean and the quadratic form is noncentral.

For example, in the fixed-design example, the slope increment satisfies $D\mu=0$ under the intercept-only mean. If instead $\mu=5u_1+2u_2$, then $D\mu=2u_2\ne0$, and $Y'DY/\sigma^2$ is noncentral chi-square with one degree of freedom and noncentrality parameter $\mu'D\mu/\sigma^2=4/\sigma^2$. Its distribution is no longer the central $\chi^2_1$ asserted under the null.

Key takeaway: Condition on the design, verify Gaussian spherical errors in that conditional distribution, check that the projection matrices are orthogonal, and check that each mean component is annihilated before claiming a central chi-square distribution. The degrees of freedom are the projection ranks.

Check Your Understanding

Use the assumptions and projection checks from this tutorial to answer the following.

For a projection $A$, what condition on the mean makes $Y'AY$ equal to $\varepsilon'A\varepsilon$?
Why does the rank of a projection give the degrees of freedom in its chi-square sum of squares?
In a nested sequence of model spaces, what does $P_j-P_{j-1}$ represent?
Which conditional assumption on the errors lets you apply the fixed-design theorem after observing a random design?
If conditional degrees of freedom vary with the design, what should you expect for the unconditional distribution?