Wishart Matrices as Gaussian Gram Matrices

From Gaussian Vectors to a Random Matrix

For a univariate normal sample, sums of squared centered observations lead to familiar chi-square distributions. In several dimensions, the corresponding object is a matrix: instead of adding squared numbers, we add outer products of random vectors. This matrix records both sums of squares in individual coordinates and sums of cross-products between coordinates.

The Wishart distribution describes the resulting random matrix when the vectors are independent and multivariate normal. Its construction is also a Gaussian Gram-matrix construction: entries of the matrix are inner-product sums formed from columns of Gaussian observations.

Definition: Let $Z_1,\ldots,Z_\nu$ be independent random vectors, each with distribution $N_p(0,\Sigma)$, where $\Sigma$ is a $p\times p$ positive-definite covariance matrix and $\nu$ is a positive integer. The random matrix $$ W=\sum_{i=1}^{\nu} Z_i Z_i^T $$ has a Wishart distribution with $\nu$ degrees of freedom and scale matrix $\Sigma$, written $W\sim W_p(\nu,\Sigma)$. The matrix $\Sigma$ is called the scale parameter in Wishart terminology; in this Gaussian construction it is the covariance matrix of each vector $Z_i$.

Each outer product $Z_iZ_i^T$ is $p\times p$. If $Z_i=(Z_{i1},\ldots,Z_{ip})^T$, its $(j,k)$ entry is $Z_{ij}Z_{ik}$. Thus $W_{jj}$ is the sum of squared observations in coordinate $j$, while $W_{jk}$, for $j\ne k$, is the sum of the cross-products between coordinates $j$ and $k$. The matrix is symmetric because every outer product is symmetric, and it is positive semidefinite because for any $a\in\mathbb R^p$, $$ a^TWa=\sum_{i=1}^{\nu}a^TZ_iZ_i^Ta =\sum_{i=1}^{\nu}(a^TZ_i)^2\geq 0. $$

The degrees of freedom $\nu$ count the number of independent Gaussian vectors being added. They are not the dimension $p$, which counts the coordinates in each vector. The scale matrix describes the covariance pattern of each vector before the outer products are formed. This definition uses integer degrees of freedom because it is based directly on a finite sum of vectors; extensions beyond this construction are not needed here.

The Gaussian Gram-Matrix View

Put the Gaussian vectors into the columns of a $p\times \nu$ matrix $A=[Z_1\ \cdots\ Z_\nu]$. Then $$ W=AA^T. $$ The $(j,k)$ entry of $AA^T$ is the inner product of row $j$ of $A$ with row $k$ of $A$. Equivalently, if the vectors are arranged as rows of a $\nu\times p$ matrix $Y=A^T$, then $W=Y^TY$, the Gram matrix of the $p$ columns of $Y$. Its entries are sums of products across the $\nu$ independent Gaussian cases.

There is a useful distinction in the two matrix products. $AA^T$ is the $p\times p$ matrix of coordinate cross-products and is the Wishart matrix in this definition. The other product, $A^TA$, is a $\nu\times\nu$ matrix of inner products between the $\nu$ Gaussian vectors. Although both products are Gram matrices in the appropriate orientation, they have different dimensions and represent different collections of inner products.

Worked Example: Computing a Gaussian Cross-Product Matrix

Let $Z_1,Z_2,Z_3$ be independent $N_2(0,I_2)$ vectors. Their sum of outer products has distribution $W_2(3,I_2)$. To see the matrix calculation, arrange three vector values as columns of $$ A= \begin{pmatrix} 1&0&1\\ 0&2&1 \end{pmatrix}. $$

The resulting cross-product matrix is $$ AA^T= \begin{pmatrix} 1&0&1\\ 0&2&1 \end{pmatrix} \begin{pmatrix} 1&0\\ 0&2\\ 1&1 \end{pmatrix} = \begin{pmatrix} 2&1\\ 1&5 \end{pmatrix}. $$

For example, the off-diagonal entry is $1(0)+0(2)+1(1)=1$, the sum of the three coordinate cross-products. The diagonal entries are $1^2+0^2+1^2=2$ and $0^2+2^2+1^2=5$. The determinant is $2(5)-1(1)=9>0$, consistent with this particular matrix being positive definite. The numerical matrix illustrates the cross-product operation; the Wishart law describes the random matrix when the columns are independent Gaussian vectors, not this one fixed numerical outcome.

Degrees of Freedom and Scale in Simple Cases

The scalar case makes the role of scale especially transparent. When $p=1$, each $Z_i$ is a univariate normal random variable, and an outer product is just a square. Consequently, the Wishart construction reduces to the usual sum of squared standardized normal variables.

Worked Example: The One-Dimensional Wishart Case

Suppose $Z_1,\ldots,Z_5$ are independent $N(0,4)$ random variables and set $W=\sum_{i=1}^5 Z_i^2$. Write $Z_i=2U_i$, where the $U_i$ are independent $N(0,1)$. Then $$ W=4\sum_{i=1}^5 U_i^2. $$

By the definition of the chi-square distribution, $\sum_{i=1}^5 U_i^2\sim\chi^2_5$. Therefore $W\sim W_1(5,4)$, and $W/4\sim\chi^2_5$. The five independent squared terms give the degrees of freedom, while the factor $4$ is the scale inherited from the variance of each $Z_i$. Since each squared term has expectation $4$, $E[W]=5(4)=20$.

For a diagonal covariance matrix, the coordinates also give a direct interpretation. If $\Sigma=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_p^2)$, then the coordinates of each $Z_i$ are independent normal variables. The diagonal entry $W_{jj}$ is a sum of $\nu$ squares with coordinate variance $\sigma_j^2$, while the off-diagonal entries still record cross-products across the same vectors. When $\Sigma$ has nonzero off-diagonal entries, those dependencies are built into the Gaussian vectors and therefore into the cross-product matrix.

Worked Example: Diagonal Scale with Two Coordinates

Let $Z_1,\ldots,Z_4$ be independent $N_2(0,\Sigma)$ vectors, where $\Sigma=\operatorname{diag}(1,9)$. For each vector, its first coordinate has variance $1$, its second has variance $9$, and the two coordinates are independent. The sum $W=\sum_{i=1}^4 Z_iZ_i^T$ has distribution $W_2(4,\Sigma)$.

The first diagonal entry is $W_{11}=\sum_{i=1}^4 Z_{i1}^2$, so $W_{11}\sim\chi^2_4$. The second is $W_{22}=\sum_{i=1}^4 Z_{i2}^2$; after dividing each coordinate by $3$, this gives $W_{22}/9\sim\chi^2_4$. The off-diagonal entry is $W_{12}=\sum_{i=1}^4 Z_{i1}Z_{i2}$. Each product has expectation zero because the two coordinates are independent and centered. In particular, the scale $9$ on the second coordinate does not mean the second diagonal entry is a single variance value: it is the sum of four squared observations at that scale.

A Basic Mean Result

Theorem: If $W\sim W_p(\nu,\Sigma)$ under the Gaussian-sum definition, then $E[W]=\nu\Sigma$, with expectation taken entry by entry.

Proof. For each $i$, the $(j,k)$ entry of $Z_iZ_i^T$ is $Z_{ij}Z_{ik}$. Since $E[Z_i]=0$ and $\operatorname{Cov}(Z_i)=\Sigma$, the definition of covariance gives $$ E[Z_{ij}Z_{ik}]=\operatorname{Cov}(Z_{ij},Z_{ik})=\Sigma_{jk}. $$ The sum defining $W$ is finite, so linearity of expectation applies entry by entry: $$ E[W_{jk}]=\sum_{i=1}^{\nu}E[Z_{ij}Z_{ik}] =\sum_{i=1}^{\nu}\Sigma_{jk} =\nu\Sigma_{jk}. $$ This holds for every pair $j,k$, hence $E[W]=\nu\Sigma$. $\square$

This result is another way to interpret the parameters: increasing the number of independent vectors multiplies the expected cross-product matrix by $\nu$, while the covariance pattern is given by $\Sigma$. It does not say that an observed $W$ equals $\nu\Sigma$; the equality is about its expectation.

Connection to Sample Covariance

For independent observations $X_1,\ldots,X_n\sim N_p(\mu,\Sigma)$, the uncentered sum about the known mean is $$ \sum_{i=1}^n (X_i-\mu)(X_i-\mu)^T. $$ It is a Wishart matrix with $n$ degrees of freedom and scale $\Sigma$, directly by the definition. When the mean is estimated by the sample mean, centering removes one sample-direction from the cross-product sum. The resulting sample covariance matrix uses divisor $n-1$, so its Wishart degrees of freedom are $n-1$, not $n$.

The following derivation makes the connection precise. Write the observations as rows of an $n\times p$ matrix $X$, let $\mathbf 1$ be the $n$-vector of ones, and let $H=I_n-\mathbf 1\mathbf 1^T/n$ be the centering matrix. The usual sample covariance matrix is $S=X^THX/(n-1)$.

Theorem: If $X_1,\ldots,X_n$ are independent $N_p(\mu,\Sigma)$ observations and $S=X^THX/(n-1)$, then $(n-1)S\sim W_p(n-1,\Sigma)$.

Proof. Choose an $n\times(n-1)$ matrix $U$ whose columns are an orthonormal basis for the subspace perpendicular to $\mathbf 1$. Thus $U^TU=I_{n-1}$ and $UU^T=H$. Write $X=\mathbf 1\mu^T+E$, where the rows of $E$ are independent $N_p(0,\Sigma)$ vectors. Since $U^T\mathbf 1=0$, $$ U^TX=U^TE,\qquad X^THX=X^TUU^TX=(U^TX)^T(U^TX). $$ The rows of $U^TE$ are jointly Gaussian. Their means are zero, and the covariance between transformed rows $r$ and $s$, for any coordinates $j,k$, is $$ \operatorname{Cov}\big((U^TE)_{rj},(U^TE)_{sk}\big) =\left(\sum_{i=1}^n U_{ir}U_{is}\right)\Sigma_{jk} =(U^TU)_{rs}\Sigma_{jk}. $$ This equals $\Sigma_{jk}$ when $r=s$ and zero when $r\ne s$. Jointly Gaussian subvectors with zero cross-covariance are independent, so the $n-1$ rows of $U^TE$ are independent $N_p(0,\Sigma)$ vectors. Their cross-product sum is therefore Wishart by the definition, with $n-1$ degrees of freedom and scale $\Sigma$. Since $X^THX=(n-1)S$, the result follows. $\square$

Worked Example: Degrees of Freedom After Estimating the Mean

Suppose $n=4$ independent observations are drawn from $N_2(\mu,\Sigma)$, where $$ \Sigma= \begin{pmatrix} 4&1\\ 1&2 \end{pmatrix}. $$ The sample covariance matrix $S$, calculated using divisor $n-1=3$, satisfies $3S\sim W_2(3,\Sigma)$. The number of coordinates is $p=2$, but the degrees of freedom are $3$ because centering the four observations by their estimated mean leaves $n-1=3$ independent Gaussian directions in the cross-product construction.

For an arithmetic illustration, consider observed vectors $(1,0)^T,(0,1)^T,(-1,0)^T,(0,-1)^T$. Their sample mean is $(0,0)^T$, and their cross-product sum is $$ \sum_{i=1}^4 X_iX_i^T= \begin{pmatrix} 2&0\\ 0&2 \end{pmatrix}. $$ Thus the observed sample covariance is $S=\frac{1}{3}\begin{pmatrix}2&0\\0&2\end{pmatrix}$, and indeed $3S$ is the centered cross-product matrix. The distributional claim concerns repeated samples from the stated Gaussian model; this calculation illustrates how the divisor and degrees of freedom line up.

Common Mistake and Why the Construction Matters

A common mistake is to say that the sample covariance itself has a Wishart distribution with scale $\Sigma$ and $n-1$ degrees of freedom. That omits the divisor in the definition of sample covariance. The Wishart object is the unscaled cross-product $(n-1)S$; dividing it by $n-1$ changes its scale. Likewise, if the population mean is known and the sum is centered at that known value, the construction uses $n$ independent vectors rather than the $n-1$ directions left after estimating the mean.

The Gaussian Gram-matrix viewpoint is useful because it translates covariance questions into cross-product questions. It identifies exactly which vectors are being summed, clarifies why the degrees of freedom depend on the number of independent Gaussian directions, and keeps scale separate from normalization. The density of the Wishart distribution requires additional calculations; the defining Gaussian construction already gives its meaning and its connection to familiar sample covariance matrices.

Check Your Understanding

Use the Gaussian-sum definition to answer each question.

If $Z_1,\ldots,Z_7$ are independent $N_3(0,\Sigma)$ vectors, what are the dimension, degrees of freedom, and scale matrix of their sum of outer products?
In the matrix arrangement $A=[Z_1\ \cdots\ Z_\nu]$, which product is the $p\times p$ Wishart cross-product matrix, and what does its $(j,k)$ entry sum?
For $Z_i\sim N(0,16)$ independently for $i=1,\ldots,6$, what is the distribution of $\sum_i Z_i^2/16$?
For a Gaussian sample of size $n$ with unknown mean, which matrix has a Wishart distribution: $S$ or $(n-1)S$? State its degrees of freedom and scale.
Explain why the expected Wishart matrix is $\nu\Sigma$, rather than just $\Sigma$.