Idempotent Matrices and Chi-Square Laws

Why Idempotence Produces a Chi-Square Law

A Gaussian quadratic form can assign different weights to different orthogonal directions. When its matrix is symmetric and idempotent, those weights have a particularly simple structure: each direction contributes either nothing or the square of one standard normal coordinate. The number of contributing directions determines the chi-square degrees of freedom.

The conclusion depends on both conditions. Symmetry makes the matrix orthogonally diagonalizable; idempotence restricts its eigenvalues to zero and one. Together with the rotational invariance of a standard normal vector, these facts give an exact distributional result.

Definition: A real $n\times n$ matrix $P$ is symmetric idempotent if $P'=P$ and $P^2=P$, where $P'$ denotes the transpose. The rank of $P$, written $\operatorname{rank}(P)$, is the dimension of its image $\{Px:x\in\mathbb R^n\}$. A chi-square random variable with positive integer $r$ degrees of freedom, written $\chi^2_r$, is the sum of the squares of $r$ independent $N(0,1)$ random variables.

In the main result, $Z\sim N_n(0,I_n)$, meaning that $Z$ is a standard $n$-dimensional normal vector with mean zero and identity covariance matrix. For $r=0$, we use the convention that $\chi^2_0$ is the constant zero. This convention includes the zero matrix case.

Orthogonal Changes of Coordinates

An orthogonal matrix $Q$ satisfies $Q'Q=QQ'=I_n$. Multiplying a standard normal vector by $Q'$ changes its coordinates but not its distribution. This is the distributional fact that allows us to choose coordinates adapted to $P$.

Lemma: If $Z\sim N_n(0,I_n)$ and $Q$ is an $n\times n$ orthogonal matrix, then $Q'Z\sim N_n(0,I_n)$.

Proof. The density of $Z$ at $z\in\mathbb R^n$ is $(2\pi)^{-n/2}\exp(-z'z/2)$. Set $Y=Q'Z$, so $Z=QY$. The absolute value of the Jacobian determinant of this change of variables is $|\det(Q)|=1$, since $Q'Q=I_n$. Also, $(Qy)'(Qy)=y'y$. Therefore the density of $Y$ at $y$ is

(2\pi)^{-n/2}\exp\!\left(-\frac{(Qy)'(Qy)}{2}\right)|\det(Q)| =(2\pi)^{-n/2}\exp\!\left(-\frac{y'y}{2}\right).

This is the density of $N_n(0,I_n)$, proving the lemma. $\square$

In particular, the coordinates of $Q'Z$ are independent $N(0,1)$ variables: its joint density factors into the product of the $n$ standard normal densities. The independence and standard normal distributions of these coordinates will identify the quadratic form below.

Main Result and Proof

Theorem: Let $Z\sim N_n(0,I_n)$, and let $P$ be a real symmetric idempotent $n\times n$ matrix. Then $$ Z'PZ\sim\chi^2_r,\qquad r=\operatorname{rank}(P). $$

Proof. By the spectral theorem for real symmetric matrices, there is an orthogonal matrix $Q$ such that $Q'PQ$ is diagonal. Write its diagonal entries as $\lambda_1,\ldots,\lambda_n$. Since $P^2=P$, the diagonal matrix also satisfies

$$ (Q'PQ)^2=Q'PQQ'PQ=Q'P^2Q=Q'PQ. $$

Consequently, each diagonal entry obeys $\lambda_i^2=\lambda_i$, so $\lambda_i$ is either zero or one. Permuting the eigenvectors if necessary, suppose exactly $r$ entries equal one and the rest equal zero. The rank of a diagonal matrix is the number of its nonzero diagonal entries. Multiplication on either side by an invertible matrix does not change rank, so $\operatorname{rank}(P)=\operatorname{rank}(Q'PQ)=r$.

Set $Y=Q'Z$. By the lemma, $Y\sim N_n(0,I_n)$, with independent standard normal coordinates. Since $Z=QY$, the quadratic form becomes

Z'PZ=Y'Q'PQY=\sum_{i=1}^{r}Y_i^2.

If $r\geq 1$, the right-hand side is a sum of squares of $r$ independent $N(0,1)$ variables, which is $\chi^2_r$ by definition. If $r=0$, every diagonal entry is zero, so $Q'PQ=0$, hence $P=0$ and $Z'PZ=0$. This agrees with the convention for $\chi^2_0$. Therefore the stated distribution holds in all cases. $\square$

Worked Applications

Worked Example: A One-Dimensional Projection in Two Coordinates

Let $P=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$ and $Z=(Z_1,Z_2)'\sim N_2(0,I_2)$. Find the distribution of $Z'PZ$.

First, $P'=P$, so $P$ is symmetric. Direct multiplication gives

P^2=\frac14 \begin{pmatrix}1&1\\1&1\end{pmatrix} \begin{pmatrix}1&1\\1&1\end{pmatrix} =\frac14\begin{pmatrix}2&2\\2&2\end{pmatrix} =P.

Its image consists of multiples of $(1,1)'$, so its rank is one. Computing the quadratic form directly,

Z'PZ =\frac12(Z_1+Z_2)^2 =\left(\frac{Z_1+Z_2}{\sqrt{2}}\right)^2.

The theorem gives $Z'PZ\sim\chi^2_1$. The calculation also shows the contributing standard normal coordinate explicitly: $(Z_1+Z_2)/\sqrt{2}$ is standard normal.

Worked Example: Selecting Two Coordinates

Let $P=\operatorname{diag}(1,1,0)$, and let $Z=(Z_1,Z_2,Z_3)'\sim N_3(0,I_3)$. The matrix is symmetric, and $P^2=\operatorname{diag}(1,1,0)=P$. Its image is the two-dimensional set of vectors $(a,b,0)'$, so its rank is two. The quadratic form is

$$ Z'PZ=Z_1^2+Z_2^2+0Z_3^2. $$

The first two coordinates are independent $N(0,1)$ variables. Thus this sum of two squares has distribution $\chi^2_2$, as the theorem predicts. The third coordinate contributes nothing because its direction corresponds to a zero eigenvalue.

Worked Example: The Centered Sum of Squares for Three Observations

Let $\mathbf{1}=(1,1,1)'$, and define $P=I_3-\mathbf{1}\mathbf{1}'/3$. For $Z=(Z_1,Z_2,Z_3)'\sim N_3(0,I_3)$, show that the centered sum of squares has a $\chi^2_2$ distribution.

Write $J=\mathbf{1}\mathbf{1}'$. Since $\mathbf{1}'\mathbf{1}=3$, multiplication gives $J^2=3J$. The matrix $P=I_3-J/3$ is symmetric, and

P^2=I_3-\frac{2J}{3}+\frac{J^2}{9} =I_3-\frac{2J}{3}+\frac{J}{3} =I_3-\frac{J}{3} =P.

For any vector $x$, $Px=x-\mathbf{1}(\mathbf{1}'x)/3$, so its coordinates sum to zero. Conversely, if the coordinates of $x$ sum to zero, then $Jx=0$ and $Px=x$. The image is therefore the plane of vectors whose coordinates sum to zero, which has dimension two. Hence $\operatorname{rank}(P)=2$.

Let $\bar Z=(Z_1+Z_2+Z_3)/3$. The coordinates of $PZ$ are $Z_i-\bar Z$, so

Z'PZ=\sum_{i=1}^{3}(Z_i-\bar Z)^2.

By the theorem, this centered sum of squares has a $\chi^2_2$ distribution. The degrees of freedom here follow from the rank of this particular matrix; no general result about sample variances is required.

What the Conditions Do—and Do Not—Guarantee

The theorem is useful because it identifies an exact reference distribution from the matrix structure alone. Once symmetry and idempotence are verified, the only additional quantity needed for the degrees of freedom is the rank. This makes the result applicable to coordinate selection, orthogonal projections, and residual or contrast sums of squares when their matrices meet the stated conditions.

A frequent mistake is to treat symmetry by itself as sufficient. Symmetry permits an orthogonal diagonalization, but a symmetric matrix can have eigenvalues other than zero and one. For example, $A=\operatorname{diag}(1,2)$ is symmetric, but $Z'AZ=Z_1^2+2Z_2^2$. The coefficient 2 does not disappear under a coordinate change, so this is not the sum of two unweighted standard normal squares. Idempotence is what forces every eigenvalue to be zero or one.

A second mistake is to use the matrix dimension $n$ as the degrees of freedom automatically. Directions associated with zero eigenvalues contribute zero to the form. The degrees of freedom count only the nonzero directions, namely $\operatorname{rank}(P)$. For instance, in the coordinate-selection example, $n=3$ but the rank and degrees of freedom are both two.

Key takeaway: For $Z\sim N_n(0,I_n)$, a symmetric idempotent matrix has only zero and one eigenvalues. Orthogonal coordinates therefore reduce $Z'PZ$ to a sum of exactly $\operatorname{rank}(P)$ independent standard normal squares.

Check Your Understanding

For each question, identify which matrix property or distributional fact is relevant.

Why does idempotence force every eigenvalue of a symmetric matrix $P$ to be either zero or one?
If $P=\operatorname{diag}(1,0,1,0)$, what is the distribution of $Z'PZ$ for $Z\sim N_4(0,I_4)$? Give the degrees of freedom and explain them.
What role does orthogonal invariance of the standard normal vector play in the proof?
Why does $A=\operatorname{diag}(1,2)$ not satisfy the theorem’s hypotheses, and what is $Z'AZ$ for $Z=(Z_1,Z_2)'$?
For $P=I_3-\mathbf{1}\mathbf{1}'/3$, explain why its centered sum of squares has two, rather than three, degrees of freedom.