Wishart Degrees Of Freedom And Rank

Why Degrees of Freedom and Rank Are Different Questions

A scatter matrix summarizes variation in several coordinates at once. Its degrees of freedom record how many independent Gaussian vector contributions enter its construction; its rank records how many independent directions the realized matrix spans. Those numbers need not agree. In particular, a scatter matrix can have positive degrees of freedom and still be singular when there are too few Gaussian contributions relative to the dimension.

The distinction matters in multivariate analysis because positive definiteness and singularity have different consequences. A positive-definite scatter matrix has a nonzero quadratic form in every nonzero direction. A singular scatter matrix has at least one direction with zero observed scatter. We derive both the degrees of freedom and the almost-sure rank from the Gaussian sampling construction.

Wishart Construction and Degrees of Freedom

Throughout, let \(p\) be the dimension and let \(\Sigma\) be a positive-definite \(p\times p\) covariance matrix. For a positive integer \(\nu\), take independent vectors \(Z_1,\ldots,Z_\nu\), each with distribution \(N_p(0,\Sigma)\), and form their sum of outer products.

W=\sum_{j=1}^{\nu} Z_jZ_j^{\mathsf T}.

Definition: The matrix \(W\) has a Wishart distribution with \(\nu\) degrees of freedom and scale matrix \(\Sigma\), written \(W\sim W_p(\nu,\Sigma)\), if it has the Gaussian construction above. In this sampling construction, the degrees of freedom \(\nu\) count the independent Gaussian vectors contributing outer products. They do not, by themselves, specify the rank of \(W\).

To connect this definition to sample scatter, arrange independent \(N_p(0,\Sigma)\) vectors as columns of a matrix \(X\). Scatter about a known mean uses every column. Scatter about the sample mean instead removes the component in the direction of the all-ones vector, leaving one fewer independent Gaussian contribution.

Result: If \(X_1,\ldots,X_n\) are independent \(N_p(\mu,\Sigma)\) vectors with positive-definite \(\Sigma\), then \( \sum_{i=1}^n (X_i-\mu)(X_i-\mu)^{\mathsf T}\sim W_p(n,\Sigma)\). The centered sample scatter \( \sum_{i=1}^n (X_i-\bar X)(X_i-\bar X)^{\mathsf T}\) has distribution \(W_p(n-1,\Sigma)\) when \(n\geq 2\), where \(\bar X=n^{-1}\sum_{i=1}^n X_i\).

Here is the reason for the difference in degrees of freedom. Write \(Z\) for the \(p\times n\) matrix whose \(i\)th column is \(X_i-\mu\), and let \(H=I_n-\mathbf{1}\mathbf{1}^{\mathsf T}/n\), where \(\mathbf{1}\) is the \(n\)-vector of ones. The centered scatter is \(ZHZ^{\mathsf T}\). The matrix \(H\) is the orthogonal projection onto the subspace perpendicular to \(\mathbf{1}\), so its rank is \(n-1\). Choose an orthogonal matrix \(Q\) whose first column is \(\mathbf{1}/\sqrt n\) and whose remaining columns form an orthonormal basis for that perpendicular subspace. Then \(Q^{\mathsf T}HQ=\operatorname{diag}(0,I_{n-1})\).

After standardizing the rows of \(Z\) by \(\Sigma^{-1/2}\), the resulting matrix has independent standard normal entries. Multiplying it on the right by \(Q\) preserves that joint standard normal distribution: each row is an isotropic standard normal vector, and different rows remain independent. Thus, in the rotated coordinates, the first column is removed by \(H\), while the other \(n-1\) columns are independent \(N_p(0,\Sigma)\) vectors after restoring the covariance. Their outer products give the claimed \(W_p(n-1,\Sigma)\) construction. For scatter about the known \(\mu\), no projection removes a column, so there are \(n\) contributions.

Almost-Sure Rank of a Wishart Matrix

Let \(A\) be the \(p\times \nu\) matrix with columns \(Z_1,\ldots,Z_\nu\). Then \(W=AA^{\mathsf T}\). The key fact is that a matrix of independent Gaussian entries has the largest rank permitted by its dimensions with probability one.

Theorem: If \(W\sim W_p(\nu,\Sigma)\) is constructed from independent \(N_p(0,\Sigma)\) vectors with positive-definite \(\Sigma\), then \( \operatorname{rank}(W)=\min(p,\nu)\) almost surely. Consequently, \(W\) is positive definite almost surely if and only if \(\nu\geq p\). If \(0<\nu<p\), then \(W\) is positive semidefinite and singular almost surely, with nullity \(p-\nu\).

Proof. Write each \(Z_j=\Sigma^{1/2}G_j\), where the columns \(G_j\) of a \(p\times \nu\) matrix \(G\) are independent \(N_p(0,I_p)\). Since \(\Sigma^{1/2}\) is invertible, multiplying by it does not change rank. Also, \(W=\Sigma^{1/2}GG^{\mathsf T}\Sigma^{1/2}\). For every \(x\in\mathbb{R}^p\),

x^{\mathsf T}GG^{\mathsf T}x =\lVert G^{\mathsf T}x\rVert^2, \qquad \operatorname{null}(GG^{\mathsf T})=\operatorname{null}(G^{\mathsf T}).

The null spaces in the equality have the same dimension, so \(\operatorname{rank}(GG^{\mathsf T})=\operatorname{rank}(G)\). It remains to establish the rank of \(G\). Let \(k=\min(p,\nu)\). Consider the first \(k\) columns of \(G\). For each successive column among these, condition on the preceding columns. If the preceding columns are linearly independent, their span is a fixed proper subspace of \(\mathbb{R}^p\), because before the \(k\)th column there are fewer than \(p\) preceding columns. A standard Gaussian vector has a density with respect to \(p\)-dimensional Lebesgue measure, and a proper linear subspace has Lebesgue measure zero. Therefore the next column lies in the preceding span with conditional probability zero. Induction shows that the first \(k\) columns are linearly independent almost surely. The rank cannot exceed either the number of rows \(p\) or the number of columns \(\nu\), so \(\operatorname{rank}(G)=k\) almost surely. The rank conclusion for \(W\) follows. Finally, \(W\) is positive semidefinite by its outer-product construction, and a symmetric positive-semidefinite matrix is positive definite exactly when its rank is \(p\). This proves the theorem. \(\square\)

For centered sample scatter the theorem applies with \(\nu=n-1\), not \(n\). Thus, for \(n\geq2\),

\operatorname{rank}\!\left( \sum_{i=1}^n (X_i-\bar X)(X_i-\bar X)^{\mathsf T} \right) =\min(p,n-1) \quad\text{almost surely}.

It follows that centered scatter is positive definite almost surely precisely when \(n-1\geq p\), or equivalently \(n\geq p+1\). Dividing this scatter by \(n-1\), as in the usual sample covariance matrix, does not change its rank when \(n\geq2\).

Worked Consequences

Worked Example: A Three-Dimensional Scatter With Too Few Contributions

Let \(p=3\), \(\Sigma=I_3\), and \(\nu=2\). For a Wishart matrix \(W_3(2,I_3)\), the degrees of freedom are \(2\), and the rank theorem gives \(\operatorname{rank}(W)=\min(3,2)=2\) almost surely. The matrix is therefore singular almost surely, despite having positive degrees of freedom.

One possible pair of Gaussian-vector values is \(Z_1=(1,0,1)^{\mathsf T}\) and \(Z_2=(0,1,1)^{\mathsf T}\). The resulting outer-product sum is

W=Z_1Z_1^{\mathsf T}+Z_2Z_2^{\mathsf T} =\begin{pmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{pmatrix}.

The third row is the sum of the first two, and \(\det(W)=1(2-1)+1(0-1)=0\). More explicitly, \(W(-1,-1,1)^{\mathsf T}=0\), so the matrix has a nonzero null vector. This particular pair illustrates the rank mechanism; the theorem says rank two holds almost surely for the Gaussian model, not that these specific values are typical.

Worked Example: A Positive-Definite Wishart Matrix

Take \(p=2\), \(\nu=3\), and \(\Sigma=\operatorname{diag}(4,1)\). The rank theorem gives rank \(\min(2,3)=2\) almost surely, so \(W\) is positive definite almost surely. To check the calculation with concrete vector values, use \(G\) with columns \((1,0)^{\mathsf T}\), \((0,1)^{\mathsf T}\), and \((1,1)^{\mathsf T}\). Since \(\Sigma^{1/2}=\operatorname{diag}(2,1)\), the corresponding \(Z\) matrix is

Z=\Sigma^{1/2}G =\begin{pmatrix} 2&0&2\\ 0&1&1 \end{pmatrix}, \qquad W=ZZ^{\mathsf T} =\begin{pmatrix} 8&2\\ 2&2 \end{pmatrix}.

Its determinant is \(8\cdot2-2\cdot2=12>0\), and its first diagonal entry is \(8>0\). For a symmetric \(2\times2\) matrix, these two inequalities establish positive definiteness. The chosen \(G\) is a concrete full-row-rank realization; the almost-sure claim comes from the theorem.

Worked Example: Centering a Small Gaussian Sample

Suppose \(n=4\) observations are independent \(N_4(\mu,\Sigma)\), with \(\Sigma\) positive definite, and let \(S=\sum_{i=1}^4(X_i-\bar X)(X_i-\bar X)^{\mathsf T}\). Centering removes one of the four sample-coordinate directions, so the degrees of freedom are \(n-1=3\) and \(S\sim W_4(3,\Sigma)\). Its almost-sure rank is

\operatorname{rank}(S)=\min(4,3)=3.

Thus \(S\) is singular almost surely, with nullity \(4-3=1\). The sample covariance \(S/3\) has the same rank. If the sample size instead were \(n=5\), the centered scatter would have \(5-1=4\) degrees of freedom and rank \(\min(4,4)=4\) almost surely, so it would be positive definite. The threshold is \(n=p+1=5\), not \(n=p\), because centering costs one degree of freedom.

Interpreting the Threshold and Avoiding a Common Error

The rank condition has a geometric interpretation. Each vector \(Z_j\) contributes an outer product whose range lies in the direction spanned by \(Z_j\). The sum of \(\nu\) such matrices cannot span more than \(\nu\) directions, and a \(p\)-dimensional matrix cannot have rank greater than \(p\). Gaussian sampling ensures that the available directions are in general position with probability one, producing exactly the smaller of those two limits.

A frequent mistake is to reason that \(W_p(\nu,\Sigma)\) has rank \(\nu\) because it is a sum of \(\nu\) outer products, or to conclude it is automatically full rank because \(\Sigma\) is positive definite. The first statement misses the dimension limit when \(\nu>p\); the second overlooks the shortage of directions when \(\nu<p\). The correction is to use \(\operatorname{rank}(W)=\min(p,\nu)\) almost surely under the stated Gaussian model. For a centered sample, first replace the raw observation count by the residual count \(n-1\).

Key takeaway: Degrees of freedom count independent Gaussian contributions; rank counts the directions their scatter spans. For positive-definite \(\Sigma\), a \(W_p(\nu,\Sigma)\) matrix has rank \(\min(p,\nu)\) almost surely. Centered scatter from \(n\) observations uses \(\nu=n-1\), so it is positive definite almost surely only when \(n\geq p+1\).

Check Your Understanding

Use the Gaussian construction and rank result to answer each question.

For \(W_5(3,\Sigma)\) with positive-definite \(\Sigma\), what are its almost-sure rank and nullity? Is it positive definite?
A sample of \(n=7\) independent Gaussian vectors has dimension \(p=6\). What are the degrees of freedom and almost-sure rank of its centered scatter?
Explain why a positive-definite scale matrix does not ensure that a Wishart matrix is positive definite.
For known-mean scatter from \(n\) Gaussian observations, how many degrees of freedom are there, and what sample-size condition makes the scatter positive definite almost surely?
Why does dividing centered scatter by \(n-1\) leave its rank unchanged when \(n\geq2\)?