Interpreting Separable Row And Column Covariance

Reading Dependence Between Matrix Entries

A matrix-valued observation can have dependence in two directions at once. For example, measurements may be associated because they come from related rows, because they come from related columns, or through both relationships together. The matrix-normal model represents these two sources of dependence separately. Its defining consequence for individual entries is a product rule: the covariance between two entries is the row covariance multiplied by the column covariance.

This product is useful for interpretation as well as calculation. It tells us when entrywise covariance is positive, negative, or zero; it also gives a direct formula for entrywise correlations. But the two covariance parameters are not themselves the covariance of every pair of entries. The other dimension contributes a factor too.

Definition: Let $X$ be a $p$-by-$q$ random matrix with mean matrix $M$. Under the column-stacking convention, $X\sim MN_{p,q}(M,U,V)$ means that $\operatorname{vec}(X)$ is multivariate normal with mean $\operatorname{vec}(M)$ and covariance matrix $V\otimes U$. Here $U$ is a $p$-by-$p$ symmetric positive-definite row covariance parameter, and $V$ is a $q$-by-$q$ symmetric positive-definite column covariance parameter.

Write $X_{ij}$ for the entry in row $i$, column $j$, and $U_{ik}$, $V_{jl}$ for entries of the two covariance parameters. The mean matrix affects the entry means, but not the covariance formula below. The vectorized covariance $V\otimes U$, established in the earlier material on vectorized matrix-normal covariance, gives the exact relationship.

Theorem: If $X\sim MN_{p,q}(M,U,V)$, then for any valid row indices $i,k$ and column indices $j,l$, $$ \operatorname{Cov}(X_{ij},X_{kl})=U_{ik}V_{jl}. $$ In particular, $$ \operatorname{Var}(X_{ij})=U_{ii}V_{jj}. $$

To see the indexing, column-stacking places $X_{ij}$ at position $i+(j-1)p$ in $\operatorname{vec}(X)$. The corresponding entry of $V\otimes U$, with row index $i+(j-1)p$ and column index $k+(l-1)p$, is $V_{jl}U_{ik}$. A covariance matrix entry at those positions is precisely $\operatorname{Cov}(X_{ij},X_{kl})$, proving the formula. Taking $k=i$ and $l=j$ gives the variance formula. $\square$

From Covariance Parameters to Correlations

The variance of an entry combines its row-specific scale $U_{ii}$ with its column-specific scale $V_{jj}$. To compare dependence across entries on a common scale, divide their covariance by their standard deviations. Define the correlations associated with the two covariance parameters by $\rho^U_{ik}=U_{ik}/\sqrt{U_{ii}U_{kk}}$ and $\rho^V_{jl}=V_{jl}/\sqrt{V_{jj}V_{ll}}$. Positive definiteness ensures that the diagonal terms in these denominators are positive.

Theorem: For $X\sim MN_{p,q}(M,U,V)$, the correlation between any two entries is $$ \operatorname{Corr}(X_{ij},X_{kl})=\rho^U_{ik}\rho^V_{jl}. $$

Indeed, the standard deviations are $\sqrt{U_{ii}V_{jj}}$ and $\sqrt{U_{kk}V_{ll}}$. Substituting the covariance and variances into the definition of correlation gives

\begin{aligned} \operatorname{Corr}(X_{ij},X_{kl}) &=\frac{U_{ik}V_{jl}} {\sqrt{U_{ii}V_{jj}}\sqrt{U_{kk}V_{ll}}}\\ &=\frac{U_{ik}}{\sqrt{U_{ii}U_{kk}}} \frac{V_{jl}}{\sqrt{V_{jj}V_{ll}}} =\rho^U_{ik}\rho^V_{jl}. \end{aligned}

Thus the sign of an entrywise correlation is the sign of the product of the two component correlations. A positive relationship in one dimension and a negative one in the other produce a negative entrywise correlation; two negative component correlations produce a positive one. If either component correlation is zero, the entrywise covariance and correlation are zero.

Worked Example: Two Entries in a Two-by-Two Matrix

Suppose $X\sim MN_{2,2}(M,U,V)$, where $U=\begin{pmatrix}4&1\\1&9\end{pmatrix}$ and $V=\begin{pmatrix}1&0.5\\0.5&4\end{pmatrix}$. Both matrices are positive definite: their diagonal entries are positive, and their determinants are $35$ and $3.75$, respectively. Find the variances of $X_{11}$ and $X_{22}$, and their covariance and correlation.

The entrywise variance rule gives $\operatorname{Var}(X_{11})=U_{11}V_{11}=4(1)=4$ and $\operatorname{Var}(X_{22})=U_{22}V_{22}=9(4)=36$.

For the covariance, the row indices are $1,2$ and the column indices are also $1,2$. Therefore, $\operatorname{Cov}(X_{11},X_{22})=U_{12}V_{12}=1(0.5)=0.5$. The correlation is $0.5/(\sqrt{4}\sqrt{36})=0.5/(2\cdot6)=1/24$, approximately $0.0417$.

The two entries are positively associated, but their correlation is small. The row covariance $U_{12}=1$ alone is not their covariance: it must be multiplied by the column covariance $V_{12}=0.5$, and correlation also accounts for both entries’ variances.

Worked Example: Two Negative Component Correlations

Let $X\sim MN_{2,2}(M,U,V)$, with $U=\begin{pmatrix}1&-0.6\\-0.6&4\end{pmatrix}$ and $V=\begin{pmatrix}9&-3\\-3&4\end{pmatrix}$. These are positive definite: their determinants are $4-0.36=3.64$ and $36-9=27$, and their diagonal entries are positive. Calculate the correlation between $X_{11}$ and $X_{22}$.

First, $\operatorname{Var}(X_{11})=U_{11}V_{11}=1(9)=9$, and $\operatorname{Var}(X_{22})=U_{22}V_{22}=4(4)=16$. Their covariance is $\operatorname{Cov}(X_{11},X_{22})=U_{12}V_{12}=(-0.6)(-3)=1.8$. Hence their correlation is $1.8/(\sqrt{9}\sqrt{16})=1.8/12=0.15$.

The factorization gives the same answer. The row correlation is $-0.6/\sqrt{1\cdot4}=-0.3$; the column correlation is $-3/\sqrt{9\cdot4}=-0.5$. Their product is $(-0.3)(-0.5)=0.15$. Each dimension contributes a negative association, so together they yield a positive entrywise correlation.

What the Product Rule Reveals

The formula can be read by holding one index fixed. If two entries lie in the same row, then $i=k$, and their covariance is $U_{ii}V_{jl}$: the row scale multiplies the column covariance. If they lie in the same column, then $j=l$, and their covariance is $U_{ik}V_{jj}$: the column scale multiplies the row covariance. If both indices differ, both off-diagonal terms contribute.

This structure also explains a useful zero pattern. A zero $U_{ik}$ makes every covariance between an entry in row $i$ and an entry in row $k$ zero, regardless of the chosen columns. Similarly, a zero $V_{jl}$ makes every covariance between entries in columns $j$ and $l$ zero, regardless of the rows. A zero in either factor is enough; it is not necessary for both factors to vanish.

Worked Example: Same-Row, Same-Column, and Zero Covariance

Consider $X\sim MN_{3,2}(M,U,V)$, with $U=\begin{pmatrix}4&2&0\\2&9&-3\\0&-3&9\end{pmatrix}$ and $V=\begin{pmatrix}1&0.4\\0.4&4\end{pmatrix}$. The leading principal minors of $U$ are $4$, $32$, and $252$, so $U$ is positive definite; $V$ is positive definite because its diagonal entries are positive and its determinant is $4-0.16=3.84$.

For two entries in row 2, $\operatorname{Cov}(X_{21},X_{22})=U_{22}V_{12}=9(0.4)=3.6$. Their variances are $U_{22}V_{11}=9$ and $U_{22}V_{22}=36$, so their correlation is $3.6/(\sqrt{9}\sqrt{36})=3.6/18=0.2$.

For two entries in column 1, $\operatorname{Cov}(X_{11},X_{21})=U_{12}V_{11}=2(1)=2$. Their variances are $4$ and $9$, giving correlation $2/(\sqrt{4}\sqrt{9})=2/6=1/3$.

Finally, $\operatorname{Cov}(X_{11},X_{32})=U_{13}V_{12}=0(0.4)=0$. Their variances are $U_{11}V_{11}=4$ and $U_{33}V_{22}=36$, so their correlation is also zero. The zero comes from the row factor even though the column covariance is nonzero.

Why Separability Matters—and a Common Mistake

Separability makes a large entrywise covariance calculation manageable. Once $U$ and $V$ are known, every pairwise covariance follows from two scalar lookups and a multiplication. It also imposes a substantive modeling structure: the association between two rows is modulated in the same way across column pairs, and the association between two columns is modulated in the same way across row pairs.

A common mistake is to treat $U_{ik}$ as the covariance between $X_{ij}$ and $X_{kl}$, or to treat $V_{jl}$ as that covariance. That omits the other dimension. The correction is to identify both row indices and both column indices, then use $U_{ik}V_{jl}$. For instance, in the first example $U_{12}=1$, but the covariance between $X_{11}$ and $X_{22}$ is $1(0.5)=0.5$, not $1$.

A second interpretive caution is that covariance and correlation answer different questions. The covariance includes the scales in $U$ and $V$; the correlation is dimensionless and is the product of their standardized correlations. Consequently, large covariance does not by itself mean a strong standardized association. Always use the variance formula before interpreting the magnitude of a covariance.

Key takeaway: Under $X\sim MN_{p,q}(M,U,V)$, entrywise variance is $U_{ii}V_{jj}$, covariance is $U_{ik}V_{jl}$, and correlation is the product of the corresponding row and column correlations. Interpret both dimensions together.

Check Your Understanding

Use the entrywise product rule to answer these questions.

If $U_{ik}=0$ but $V_{jl}\ne0$, what are $\operatorname{Cov}(X_{ij},X_{kl})$ and $\operatorname{Corr}(X_{ij},X_{kl})$?
Suppose the row correlation for a pair of rows is $-0.4$ and the column correlation for a pair of columns is $0.25$. What is the correlation between the corresponding entries?
For entries in the same row, which covariance parameter determines their correlation after standardization?
If $U_{ii}=5$ and $V_{jj}=2$, what is $\operatorname{Var}(X_{ij})$?
Why is it generally incorrect to identify $U_{ik}$ alone as the covariance between $X_{ij}$ and $X_{kl}$?

Interpreting Separable Row And Column Covariance

What You'll Learn