The Wishart Law of a Normal Sample Covariance

From a Normal Sample to a Matrix Distribution

For a multivariate sample, variability is recorded not by one sum of squared deviations but by a whole matrix of sums of products. Its diagonal entries are sums of squared deviations for individual coordinates; its off-diagonal entries record cross-products between coordinates. When the observations are independent and multivariate normal, this centered matrix has an exact sampling distribution: the Wishart distribution.

The key issue is centering. If the population mean were known and the observations were summed as outer products about that known mean, there would be $n$ independent normal contributions. When the same observations are used to estimate the mean, the centered sum has $n-1$ degrees of freedom. The derivation below makes this reduction explicit and also shows when the resulting matrix is singular.

Definition: Let $Z_1,\ldots,Z_\nu$ be independent random vectors with $Z_j\sim N_p(0,\Sigma)$, where $\Sigma$ is a positive-definite $p\times p$ matrix and $\nu$ is a positive integer. The distribution of $$ W=\sum_{j=1}^{\nu} Z_jZ_j^{T} $$ is the Wishart distribution with scale matrix $\Sigma$ and $\nu$ degrees of freedom, denoted $W\sim W_p(\Sigma,\nu)$. This sum representation defines the distribution, including when $\nu<p$ and $W$ is singular.

Each outer product $Z_jZ_j^T$ is a $p\times p$ matrix. The Wishart matrix $W$ is symmetric and positive semidefinite, since for every $a\in\mathbb{R}^p$, $a^TWa=\sum_{j=1}^{\nu}(a^TZ_j)^2\geq 0$. The parameter $\nu$ counts the independent normal vectors in the defining sum; it is not the dimension $p$.

Deriving the Centered-Sample Law

Let $X_1,\ldots,X_n$ be independent observations with $X_i\sim N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite and $n\geq 2$. Write $\bar X=n^{-1}\sum_{i=1}^n X_i$, and define the centered sum-of-products matrix $W=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^T$. The sample covariance matrix with the usual $n-1$ divisor is $S=W/(n-1)$.

Theorem: Under the conditions above, the centered sum-of-products matrix satisfies $$ \sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)^T\sim W_p(\Sigma,n-1). $$ Consequently, the sample covariance matrix has the distribution of $W/(n-1)$ for $W\sim W_p(\Sigma,n-1)$.

To prove the theorem, place the observations in the columns of the $p\times n$ matrix $X=[X_1\ \cdots\ X_n]$. Choose an $n\times n$ orthogonal matrix $Q$ whose first column is $n^{-1/2}\mathbf{1}$, where $\mathbf{1}$ is the vector of $n$ ones. The other columns of $Q$ are orthonormal contrasts: each is perpendicular to $\mathbf{1}$.

Transform the observation columns.
Set $Y=XQ$, with columns $Y_1,\ldots,Y_n$. Orthogonality of $Q$ preserves the total cross-product structure while expressing it in a basis whose first direction is the sample-mean direction.

Identify the transformed columns.
Since the first column of $Q$ is $n^{-1/2}\mathbf{1}$, $Y_1=\sqrt n\,\bar X$. For $j\geq 2$, the coefficients in column $j$ sum to zero, so $E[Y_j]=0$.

Find their distributions.
The transformed columns are jointly multivariate normal. For $j,k$, their cross-covariance is $\operatorname{Cov}(Y_j,Y_k)=\sum_{i=1}^n q_{ij}q_{ik}\Sigma$, which equals $\Sigma$ if $j=k$ and zero otherwise. Thus $Y_2,\ldots,Y_n$ are independent and each has distribution $N_p(0,\Sigma)$.

Express the centered matrix using contrasts.
The centering matrix $I_n-\mathbf{1}\mathbf{1}^T/n$ is the orthogonal projection onto the contrast directions. In the $Q$ basis it is $\operatorname{diag}(0,1,\ldots,1)$, so the centered sum is $\sum_{j=2}^nY_jY_j^T$.

For completeness, the matrix identity in the final step follows directly from $$ \begin{aligned} \sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^T &=X\left(I_n-\frac{\mathbf{1}\mathbf{1}^T}{n}\right)X^T\\ &=XQ\operatorname{diag}(0,1,\ldots,1)Q^TX^T\\ &=\sum_{j=2}^nY_jY_j^T. \end{aligned} $$

There are exactly $n-1$ terms in this sum, and the preceding covariance calculation shows that they are independent $N_p(0,\Sigma)$ vectors. It therefore matches the defining construction of $W_p(\Sigma,n-1)$, proving the theorem. Centering removes the one direction parallel to $\mathbf{1}$; it does not remove one degree of freedom separately from each coordinate.

Degrees of Freedom, Scaling, and Rank

The theorem concerns the sum-of-products matrix $W$, not the covariance matrix $S$ itself. Since $S=W/(n-1)$, its distribution is a scaled Wishart distribution. The divisor changes the scale of the matrix; it does not change the degrees of freedom.

Result: For the sample covariance $S=W/(n-1)$ in the theorem, $$ E[W]=(n-1)\Sigma \qquad\text{and}\qquad E[S]=\Sigma. $$ Indeed, each defining vector $Z_j$ has $E[Z_jZ_j^T]=\Sigma$, so expectation of the sum gives $E[W]=(n-1)\Sigma$; division by $n-1$ gives the second identity.

Rank follows from the sum representation. Let $A$ be the $p\times(n-1)$ matrix with columns $Y_2,\ldots,Y_n$. Then $W=AA^T$, so $\operatorname{rank}(W)=\operatorname{rank}(A)\leq\min(p,n-1)$. Because $\Sigma$ is positive definite, $A$ has the same rank as a matrix of independent standard normal entries, after an invertible linear transformation. Such a matrix has rank $\min(p,n-1)$ with probability one: a square minor of that size is a nonzero polynomial in continuously distributed entries, and its zero set has probability zero. Thus $$ \operatorname{rank}(W)=\min(p,n-1)\quad\text{with probability one}. $$

In particular, when $n-1\geq p$, $W$ and $S$ are positive definite with probability one. When $n-1<p$, they are singular with probability one. This is a structural constraint, not a numerical accident: a sum of $n-1$ outer products cannot have rank greater than $n-1$.

Worked Example: A Bivariate Sample With Five Observations

Suppose $X_1,\ldots,X_5$ are independent with $X_i\sim N_2(\mu,\Sigma)$, where $$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ Find the law of the centered sum-of-products matrix and the expectation of the sample covariance matrix.

Here $p=2$, $n=5$, and the scale matrix is positive definite because its leading diagonal entry is $4>0$ and its determinant is $4(2)-1(1)=7>0$. The theorem gives $$ W=\sum_{i=1}^{5}(X_i-\bar X)(X_i-\bar X)^T\sim W_2(\Sigma,4). $$

The sample covariance is $S=W/4$. By the expectation calculation, $$ E[W]=4\Sigma =\begin{pmatrix}16&4\\4&8\end{pmatrix}, \qquad E[S]=\frac{1}{4}E[W] =\begin{pmatrix}4&1\\1&2\end{pmatrix} =\Sigma. $$ There are four contrast vectors, and $4\geq2$, so $W$ and $S$ are positive definite with probability one.

Worked Example: Fewer Residual Directions Than Coordinates

Let $X_1,X_2,X_3$ be independent with $X_i\sim N_3(\mu,I_3)$. Determine the Wishart degrees of freedom and the rank of their sample covariance.

Here $p=3$ and $n=3$, so $$ W=\sum_{i=1}^{3}(X_i-\bar X)(X_i-\bar X)^T\sim W_3(I_3,2), \qquad S=\frac{W}{2}. $$ The matrix $A$ of the two contrast columns has size $3\times2$, so $\operatorname{rank}(W)=\operatorname{rank}(A)\leq2$. The rank result shows equality with probability one. Therefore $S$ has rank $2$ with probability one and cannot be inverted, despite the nonsingular population covariance $I_3$. The sample has only two residual directions available to span three coordinates.

Worked Example: The One-Dimensional Chi-Square Case

Suppose $X_1,\ldots,X_6$ are independent with $X_i\sim N(3,4)$. Find the distribution of the usual sample variance $S^2$, and calculate its expectation.

With $p=1$, the Wishart matrix is a scalar. The theorem gives $$ \frac{1}{4}\sum_{i=1}^{6}(X_i-\bar X)^2\sim W_1(1,5). $$ For a one-dimensional standard normal variable, its squared value has a chi-square distribution with one degree of freedom. Thus a sum of five independent such squares is $\chi^2_5$, and $$ \frac{\sum_{i=1}^{6}(X_i-\bar X)^2}{4}\sim\chi^2_5. $$ The sample variance uses divisor $6-1=5$, so $$ S^2=\frac{1}{5}\sum_{i=1}^{6}(X_i-\bar X)^2 \ \sim\ \frac{4}{5}\chi^2_5, \qquad E[S^2]=\frac{4}{5}\cdot 5=4. $$ The mean $3$ does not appear in the centered sum's distribution; centering removes the mean direction.

Common Mistake: Confusing the Divisor With the Degrees of Freedom

A frequent error is to say that a sample of size $n$ produces a Wishart matrix with $n$ degrees of freedom because it contains $n$ observations. That count applies to an uncentered sum of outer products about a known zero mean under a zero-mean normal model. In the centered sum, the residuals satisfy $\sum_{i=1}^n(X_i-\bar X)=0$, leaving only $n-1$ independent contrast directions.

Another mistake is to assume that a positive-definite population covariance guarantees an invertible sample covariance. It does not: invertibility also requires at least $p$ residual directions, so $n-1\geq p$. If $n-1<p$, singularity holds with probability one.

Key takeaway: For independent $N_p(\mu,\Sigma)$ observations with positive-definite $\Sigma$, the centered sum-of-products matrix has law $W_p(\Sigma,n-1)$, and the usual sample covariance is that matrix divided by $n-1$. Its rank is $\min(p,n-1)$ with probability one.

Check Your Understanding

Use the sum representation and the rank condition to answer these questions.

For a sample of size $n=8$ from a $4$-dimensional normal distribution, what are the degrees of freedom of the centered Wishart matrix?
If $p=6$ and $n=4$, what is the almost-sure rank of the sample covariance matrix, and can it be inverted?
For $X_i\sim N_p(\mu,\Sigma)$, why does the degrees-of-freedom parameter not depend on the value of $\mu$?
In the univariate case with $n=10$ and variance $\sigma^2$, express the centered sum of squares divided by $\sigma^2$ as a chi-square variable.
If $S=W/(n-1)$ and $W\sim W_p(\Sigma,n-1)$, what is $E[S]$?

The Wishart Law of a Normal Sample Covariance

What You'll Learn