Covariances Among Wishart Entries

Why Entrywise Covariances Matter

A Wishart matrix records sums of products of Gaussian coordinates. Its entries are therefore related, and understanding that dependence is important when assessing the variability of individual matrix elements. For example, the uncertainty in a diagonal entry is not described by the same formula as the uncertainty in an off-diagonal entry.

The entrywise covariance formula has a simple structure: it combines two products of population covariance entries and scales them by the Wishart degrees of freedom. We derive it from the Gaussian sampling construction, then use it to calculate variances and covariances in concrete cases.

Setup and Definition

Let $p$ be the dimension and let $\nu$ be a positive integer. Suppose $X_1,\ldots,X_\nu$ are independent random vectors with distribution $N_p(0,\Sigma)$, where $\Sigma$ is a positive-definite $p\times p$ covariance matrix. The central Wishart matrix formed from these vectors is their sum of outer products.

Definition: The matrix $W=\sum_{a=1}^{\nu}X_aX_a^{\mathsf T}$ has a central Wishart distribution with $\nu$ degrees of freedom and scale matrix $\Sigma$, written $W\sim W_p(\nu,\Sigma)$. Its $(i,j)$ entry is $W_{ij}=\sum_{a=1}^{\nu}X_{ai}X_{aj}$, where $X_{ai}$ is coordinate $i$ of vector $X_a$. The matrix is symmetric, so $W_{ij}=W_{ji}$.

Here $\nu$ is an integer because we use the sampling construction directly. Indices $i,j,k,l$ range from $1$ to $p$. The entries of $\Sigma$ are $\Sigma_{ij}=\operatorname{Cov}(X_{ai},X_{aj})$, independent of the choice of row $a$. We will use the standard Gaussian fourth-moment identity and derive the entrywise Wishart result from it.

The Covariance Formula

For a single centered Gaussian vector $X\sim N_p(0,\Sigma)$, its moment-generating function is $M(t)=\exp(t^{\mathsf T}\Sigma t/2)$. The degree-four term in the Taylor expansion of this function is $(t^{\mathsf T}\Sigma t)^2/8$. Differentiating once with respect to each of $t_i,t_j,t_k,t_l$ and evaluating at zero gives the three possible pairings of the four coordinates:

E[X_iX_jX_kX_l] =\Sigma_{ij}\Sigma_{kl} +\Sigma_{ik}\Sigma_{jl} +\Sigma_{il}\Sigma_{jk}.

This identity remains valid when some indices coincide: the derivatives then produce the appropriate repeated terms. The differentiations are justified because the Gaussian moment-generating function is finite and analytic in a neighbourhood of zero.

Theorem: If $W\sim W_p(\nu,\Sigma)$ is constructed from $\nu$ independent $N_p(0,\Sigma)$ vectors, then for any $i,j,k,l$, $$ \operatorname{Cov}(W_{ij},W_{kl}) =\nu\left(\Sigma_{ik}\Sigma_{jl}+\Sigma_{il}\Sigma_{jk}\right). $$

Proof. Write $Y_a=X_{ai}X_{aj}$ and $Z_a=X_{ak}X_{al}$. Then $W_{ij}=\sum_{a=1}^{\nu}Y_a$ and $W_{kl}=\sum_{a=1}^{\nu}Z_a$. Independence of the Gaussian vectors across $a$ implies that terms from different rows have zero covariance, so

\operatorname{Cov}(W_{ij},W_{kl}) =\sum_{a=1}^{\nu}\operatorname{Cov}(Y_a,Z_a) =\nu\operatorname{Cov}(X_iX_j,X_kX_l),

where the coordinates on the right belong to any one row, which has distribution $N_p(0,\Sigma)$. By the fourth-moment identity,

\begin{aligned} \operatorname{Cov}(X_iX_j,X_kX_l) &=E[X_iX_jX_kX_l]-E[X_iX_j]E[X_kX_l]\\ &=\Sigma_{ij}\Sigma_{kl}+\Sigma_{ik}\Sigma_{jl} +\Sigma_{il}\Sigma_{jk}-\Sigma_{ij}\Sigma_{kl}\\ &=\Sigma_{ik}\Sigma_{jl}+\Sigma_{il}\Sigma_{jk}. \end{aligned}

Multiplying by $\nu$ proves the formula. $\square$

Important Special Cases

Setting $k=i$ and $l=j$ gives the variance of any entry. For a diagonal entry, set all four indices equal. For two different diagonal entries, use $i=j$ and $k=l$, with $i\ne k$.

Corollary: For $i\ne j$, $$ \operatorname{Var}(W_{ii})=2\nu\Sigma_{ii}^{\,2}, \qquad \operatorname{Cov}(W_{ii},W_{jj})=2\nu\Sigma_{ij}^{\,2}, \qquad \operatorname{Var}(W_{ij})=\nu\left(\Sigma_{ii}\Sigma_{jj}+\Sigma_{ij}^{\,2}\right). $$

Each identity follows by direct substitution in the theorem. For $\operatorname{Var}(W_{ii})$, take $i=j=k=l$; the two terms are each $\Sigma_{ii}^2$. For $\operatorname{Cov}(W_{ii},W_{jj})$, substitute $(i,i,j,j)$; both terms equal $\Sigma_{ij}^2$. For $\operatorname{Var}(W_{ij})$, substitute $(i,j,i,j)$; the terms are $\Sigma_{ii}\Sigma_{jj}$ and $\Sigma_{ij}^2$. Each result is multiplied by $\nu$. $\square$

For two possibly distinct off-diagonal entries, the general formula is often the most useful. For instance, when the entries share an index,

\operatorname{Cov}(W_{ij},W_{ik}) =\nu\left(\Sigma_{ii}\Sigma_{jk}+\Sigma_{ik}\Sigma_{ij}\right).

The two terms have different roles: one pairs the shared coordinate with itself, and the other pairs the remaining coordinates with each other. The formula also handles entries with no shared index; their covariance need not be zero if the corresponding entries of $\Sigma$ are nonzero.

Worked Example: Variances and Covariance of Diagonal Entries

Let $\nu=5$ and suppose $$ \Sigma=\begin{pmatrix}4&1.2\\1.2&9\end{pmatrix}. $$ Find the variances of $W_{11}$ and $W_{22}$, and their covariance.

For the first diagonal entry, $\Sigma_{11}=4$, so $$ \operatorname{Var}(W_{11})=2(5)(4^2)=10(16)=160. $$ For the second, $\Sigma_{22}=9$, giving $$ \operatorname{Var}(W_{22})=2(5)(9^2)=10(81)=810. $$

The covariance between the two diagonal entries depends on the square of the population covariance: $$ \operatorname{Cov}(W_{11},W_{22}) =2(5)(1.2^2)=10(1.44)=14.4. $$ The covariance is positive even though $\Sigma_{12}=1.2$ could have been negative: its sign disappears on squaring.

Worked Example: Covariance Between Two Off-Diagonal Entries

Let $\nu=4$ and take $$ \Sigma=\begin{pmatrix} 2&0.5&-0.3\\ 0.5&3&0.8\\ -0.3&0.8&5 \end{pmatrix}. $$ Find $\operatorname{Cov}(W_{12},W_{13})$, and calculate the variance of each entry as a check on their scale.

For the covariance, use $(i,j,k,l)=(1,2,1,3)$: $$ \begin{aligned} \operatorname{Cov}(W_{12},W_{13}) &=4\left(\Sigma_{11}\Sigma_{23}+\Sigma_{13}\Sigma_{21}\right)\\ &=4\left((2)(0.8)+(-0.3)(0.5)\right)\\ &=4(1.6-0.15)=4(1.45)=5.8. \end{aligned} $$

For $W_{12}$, the variance formula gives $$ \operatorname{Var}(W_{12})=4\left((2)(3)+(0.5)^2\right) =4(6.25)=25. $$ For $W_{13}$, it gives $$ \operatorname{Var}(W_{13})=4\left((2)(5)+(-0.3)^2\right) =4(10.09)=40.36. $$ The positive covariance $5.8$ is compatible with the negative value of $\Sigma_{13}$: both terms in the covariance formula contribute, and their sum is positive in this example.

Worked Example: Zero Covariance Does Not Guarantee Independence

Let $\nu=1$, $p=3$, and $\Sigma=I_3$, the $3\times3$ identity matrix. The coordinates $X_1,X_2,X_3$ are independent standard normal variables, and $W_{12}=X_1X_2$, $W_{13}=X_1X_3$. Calculate their covariance and use moments to determine whether they are independent.

The covariance formula gives $$ \operatorname{Cov}(W_{12},W_{13}) =\Sigma_{11}\Sigma_{23}+\Sigma_{13}\Sigma_{21} =(1)(0)+(0)(0)=0. $$ Also, $E[W_{12}^2]=E[X_1^2]E[X_2^2]=1$, and $E[W_{13}^2]=1$. But $$ E[W_{12}^2W_{13}^2] =E[X_1^4X_2^2X_3^2] =E[X_1^4]E[X_2^2]E[X_3^2] =(3)(1)(1)=3. $$ If $W_{12}$ and $W_{13}$ were independent, their squared values would also be independent, and the expectation of their product would be $E[W_{12}^2]E[W_{13}^2]=1$. Since $3\ne1$, the entries are not independent despite having zero covariance.

Reading the Formula Correctly

The covariance formula describes dependence among entries through the covariance structure $\Sigma$. It also makes clear that the degrees of freedom affect the scale linearly: doubling $\nu$ doubles each entrywise covariance, under the same $\Sigma$. Variances are particular covariances, so the same result supplies uncertainty calculations for both diagonal and off-diagonal elements.

A common mistake is to assume that zero covariance implies independence. That implication is not valid in general, and the preceding example demonstrates the failure even for a central Wishart matrix with identity scale. Another mistake is to treat the two covariance terms as interchangeable with a single product: their indices must be placed exactly as $\Sigma_{ik}\Sigma_{jl}+\Sigma_{il}\Sigma_{jk}$. A reliable check is to substitute the requested four indices before simplifying.

Symmetry also matters. Since $W_{ij}=W_{ji}$, these are the same random variable, not two distinct entries. Substituting either ordering into the formula must give the same covariance. For example, the variance expression for $W_{ij}$ is unchanged if $i$ and $j$ are swapped, as it must be.

Key takeaway: For a central Wishart matrix, $\operatorname{Cov}(W_{ij},W_{kl})=\nu(\Sigma_{ik}\Sigma_{jl}+\Sigma_{il}\Sigma_{jk})$. Substituting indices carefully yields diagonal and off-diagonal variances, as well as covariances between different entries; zero covariance alone does not establish independence.

Check Your Understanding

Use the entrywise covariance formula to answer each question.

For $i\ne j$, what is $\operatorname{Var}(W_{ii})$? Which entries of $\Sigma$ determine it?
Why does $\operatorname{Cov}(W_{ii},W_{jj})$ involve the square of $\Sigma_{ij}$?
Write the formula for $\operatorname{Cov}(W_{ij},W_{ik})$, where $i,j,k$ are distinct.
If $\Sigma=I_p$, what is $\operatorname{Cov}(W_{12},W_{13})$? Does that covariance establish independence?
Explain why the degrees of freedom multiply, rather than divide, the entrywise covariance formula.