Rank, Trace, and Projection Dimension

Why Projection Dimension Determines the Distribution

A symmetric idempotent matrix represents an orthogonal projection: it keeps the component of a vector in a subspace and removes the component perpendicular to that subspace. The dimension of the retained subspace is the projection’s rank. For a standard Gaussian vector, the quadratic form produced by this projection has a chi-square distribution whose degrees of freedom equal that rank.

The connection is useful because a matrix can be large while its projection dimension is small. Rather than trying to work directly with every entry of a quadratic form, we can identify the dimension of the projected subspace. For symmetric idempotent matrices, the trace and eigenvalues give that dimension directly.

Definition: A square matrix \(P\) is idempotent if \(P^2=P\). It is a symmetric idempotent matrix if, in addition, \(P'=P\), where the prime denotes transpose. Such a matrix is the orthogonal projection onto its range, \(\operatorname{range}(P)=\{Px:x\in\mathbb{R}^p\}\). Its projection dimension is \(\operatorname{rank}(P)=\dim(\operatorname{range}(P))\).

For a \(p\times p\) matrix \(P\), the trace is the sum of its diagonal entries, written \(\operatorname{tr}(P)\). The rank is a dimension, while the trace is a sum; for symmetric idempotent matrices, they agree. This agreement is special to the structure of these matrices and should not be assumed for an arbitrary matrix.

Eigenvalues, Rank, and Trace

The key fact is that idempotence restricts every eigenvalue to either zero or one. Symmetry then guarantees that the matrix has an orthonormal basis of eigenvectors. In that basis, the matrix is diagonal, with one entry for each direction it retains and zero for each direction it removes.

Theorem: If \(P\) is a real symmetric idempotent \(p\times p\) matrix, all its eigenvalues are zero or one. If exactly \(r\) eigenvalues equal one, then \(\operatorname{rank}(P)=\operatorname{tr}(P)=r\), and \(\dim(\operatorname{null}(P))=p-r\).

Proof. Let \(v\) be an eigenvector of \(P\) with eigenvalue \(\lambda\), so \(Pv=\lambda v\) and \(v\ne 0\). Idempotence gives \(P^2v=Pv\), hence \(\lambda^2v=\lambda v\). Since \(v\ne 0\), \(\lambda^2=\lambda\), so \(\lambda(\lambda-1)=0\) and \(\lambda\) is zero or one.

By the spectral theorem for real symmetric matrices, there is an orthogonal matrix \(U\) such that \(P=U\Lambda U'\), where \(\Lambda\) is diagonal and its entries are the eigenvalues of \(P\). If \(r\) entries equal one, then \(\Lambda\) has \(r\) ones and \(p-r\) zeros. Orthogonal change of coordinates preserves rank, so \(\operatorname{rank}(P)=\operatorname{rank}(\Lambda)=r\). Trace is also preserved under this change of coordinates: by \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\), \(\operatorname{tr}(P)=\operatorname{tr}(U\Lambda U')=\operatorname{tr}(\Lambda U'U)=\operatorname{tr}(\Lambda)=r\). Finally, the null space has one eigenvector direction for each zero eigenvalue, so its dimension is \(p-r\). \(\square\)

There is also a direct geometric interpretation. Every vector \(x\in\mathbb{R}^p\) can be written as \(x=Px+(I-P)x\). The first term is in the range of \(P\), and \(P(I-P)x=0\), so the second is in its null space. These spaces are perpendicular: if \(a=Pu\) and \(Pb=0\), then \(a'b=u'Pb=0\), using symmetry. Thus \(P\) retains precisely the range directions and removes the null-space directions.

Gaussian Quadratic Forms from Projection Matrices

Let \(Z\sim N_p(0,I_p)\), where \(I_p\) is the \(p\times p\) identity matrix. A standard normal vector remains standard normal after an orthogonal change of coordinates. A chi-square random variable with \(r\) degrees of freedom, written \(\chi^2_r\), is the sum of squares of \(r\) independent standard normal random variables.

Theorem: If \(Z\sim N_p(0,I_p)\) and \(P\) is a fixed symmetric idempotent \(p\times p\) matrix of rank \(r\), then \(Z'PZ\sim\chi^2_r\). When \(r=0\), the quadratic form is identically zero.

Proof. By the spectral theorem and the rank–trace result, \(P=U\Lambda U'\) for an orthogonal matrix \(U\), where \(\Lambda\) has \(r\) ones followed by \(p-r\) zeros (after ordering the eigenvectors). Set \(W=U'Z\). Since \(Z\) is standard multivariate normal and \(U\) is orthogonal, \(W\sim N_p(0,I_p)\); in particular, its coordinates are independent standard normal random variables. Therefore

Z'PZ=Z'U\Lambda U'Z=W'\Lambda W=\sum_{j=1}^{r}W_j^2\sim\chi^2_r.

If \(r=0\), the sum has no terms and equals zero. \(\square\)

This result turns a matrix question into a dimension question. To find the central chi-square degrees of freedom, establish that the matrix is symmetric and idempotent, then compute its rank. The trace can be a convenient way to obtain that rank. The theorem assumes a standard, centered Gaussian vector and a fixed symmetric idempotent matrix; dropping these conditions can change the distribution.

Worked Example: A Rank-One Projection in Four Dimensions

Let \(u=(1,2,2,0)'/3\) and \(P=uu'\). Since \(u'u=(1+4+4+0)/9=1\), \(P\) is the projection onto the line spanned by \(u\). Explicitly,

P=\frac{1}{9} \begin{pmatrix} 1&2&2&0\\ 2&4&4&0\\ 2&4&4&0\\ 0&0&0&0 \end{pmatrix}.

The matrix is symmetric. Also \(P^2=uu'uu'=u(u'u)u'=uu'=P\), so it is idempotent. Its range is the one-dimensional span of \(u\), giving rank one. The trace confirms this: \(\operatorname{tr}(P)=(1+4+4+0)/9=1\). For \(Z\sim N_4(0,I_4)\), the theorem gives \(Z'PZ\sim\chi^2_1\). Indeed, \(Z'PZ=Z'uu'Z=(u'Z)^2\), and \(u'Z\) is standard normal because \(u\) is a unit vector.

Worked Example: Complementary Projections of Different Ranks

Use the same \(P\) as above and let \(Q=I_4-P\). Then \(Q'=Q\), and \(Q^2=(I_4-P)^2=I_4-2P+P^2=I_4-P=Q\). Thus \(Q\) is also a symmetric idempotent matrix. Its trace is \(\operatorname{tr}(Q)=\operatorname{tr}(I_4)-\operatorname{tr}(P)=4-1=3\), so its rank is three.

For \(Z\sim N_4(0,I_4)\), the two quadratic forms have distributions \(Z'PZ\sim\chi^2_1\) and \(Z'QZ\sim\chi^2_3\). Their ranks differ because one projection retains a line and the other retains its three-dimensional orthogonal complement. Each distribution follows separately from the theorem; finding these marginal distributions does not, by itself, establish a claim about the joint distribution of the two forms.

Worked Example: Projection Onto a Regression Model

Consider the four design points \(x=(-3,-1,1,3)'\) and a linear regression model with an intercept and a slope. Its design matrix is \(X=[\mathbf{1},x]\), where \(\mathbf{1}=(1,1,1,1)'\). The columns are linearly independent, so \(\operatorname{rank}(X)=2\), and \(X'X=\operatorname{diag}(4,20)\). The projection onto the column space of \(X\) is \(H=X(X'X)^{-1}X'\), which gives

H= \begin{pmatrix} 7/10&2/5&1/10&-1/5\\ 2/5&3/10&1/5&1/10\\ 1/10&1/5&3/10&2/5\\ -1/5&1/10&2/5&7/10 \end{pmatrix}.

This matrix is symmetric. To check idempotence, use \(H=X(X'X)^{-1}X'\) and the invertibility of \(X'X\):

H^2=X(X'X)^{-1}(X'X)(X'X)^{-1}X'=X(X'X)^{-1}X'=H.

Its trace is \(7/10+3/10+3/10+7/10=2\), agreeing with its rank, which is the dimension of the two-column design space. Consequently, if \(Z\sim N_4(0,I_4)\), then \(Z'HZ\sim\chi^2_2\). The conclusion depends on the projection’s rank, not on the number of rows in the design matrix.

Common Mistake: Treating Trace as a General Rank Formula

For a symmetric idempotent matrix, trace equals rank because its eigenvalues are all zero or one. Symmetry alone is not enough. For example, \(A=\operatorname{diag}(2,0)\) is symmetric, has trace two, and has rank one. It is not idempotent, since \(A^2=\operatorname{diag}(4,0)\ne A\). Therefore, the rank–trace identity and the chi-square conclusion for projections cannot be applied to \(A\).

A second error is to use the ambient dimension \(p\) as the degrees of freedom automatically. A projection in \(\mathbb{R}^p\) can have any rank from zero to \(p\); its chi-square degrees of freedom are the retained dimension \(r\), not necessarily \(p\). For the identity projection, \(r=p\). For the zero matrix, \(r=0\), and the form is identically zero.

Key takeaway: For a symmetric idempotent \(P\), the eigenvalues are zeros and ones, and \(\operatorname{rank}(P)=\operatorname{tr}(P)\). If \(Z\sim N_p(0,I_p)\), then \(Z'PZ\sim\chi^2_{\operatorname{rank}(P)}\). Check symmetry and idempotence before using either conclusion.

Check Your Understanding

Use the projection structure, rather than the ambient dimension alone, to answer each question.

A symmetric idempotent \(6\times6\) matrix has trace four. What are its rank and null-space dimension?
For \(Z\sim N_5(0,I_5)\), what is the distribution of \(Z'PZ\) if \(P\) is a symmetric idempotent matrix of rank three?
Why does symmetry by itself not guarantee that the trace equals the rank?
If \(P\) is a rank-two projection in \(\mathbb{R}^5\), what is the rank of \(I_5-P\), and what is the distribution of \(Z'(I_5-P)Z\) for standard \(Z\)?
A projection matrix has rank zero. What does it do to every vector, and what value does its Gaussian quadratic form take?

Rank, Trace, and Projection Dimension

What You'll Learn