Vectorized Matrix-Normal Covariance Derivation

Why Vectorization Order Matters

A matrix-valued observation can be treated as a vector by listing its entries in a fixed order. This lets us express its distribution using familiar multivariate-normal notation, but the order is not a cosmetic choice: it determines the arrangement of the covariance matrix. For a matrix-normal variable, getting that arrangement right means keeping track of which scale matrix indexes rows and which indexes columns.

We use column-wise vectorization: list the first column from top to bottom, then the second column, and continue. Under this convention, the covariance of the resulting vector is the column scale matrix in the first Kronecker factor and the row scale matrix in the second.

Definition: For an $r\times c$ matrix $X$, $\operatorname{vec}(X)$ is the $rc$-vector obtained by stacking the columns of $X$, in order. If $X=(X_{ij})$, its entry $X_{ij}$ appears at position $i+(j-1)r$.

Let $U$ be an $r\times r$ symmetric positive-definite matrix, called the row scale, and let $V$ be a $c\times c$ symmetric positive-definite matrix, called the column scale. With these conventions, $X\sim MN_{r,c}(M,U,V)$ has the matrix-normal density

f(X)=(2\pi)^{-rc/2}|U|^{-c/2}|V|^{-r/2} \exp\!\left\{-\frac12\operatorname{tr}\!\left[V^{-1}(X-M)^{T}U^{-1}(X-M)\right]\right\}.

Here $M$ is an $r\times c$ mean matrix. Positive definiteness ensures that the inverses and determinant factors in the density exist. The purpose of this tutorial is to determine the ordinary multivariate-normal distribution of $\operatorname{vec}(X)$ and, in particular, to derive its covariance in the stated order.

The Vectorized Distribution

Theorem: If $X\sim MN_{r,c}(M,U,V)$, with $U$ and $V$ symmetric positive definite as above, then $$ \operatorname{vec}(X)\sim N_{rc}\!\left(\operatorname{vec}(M),\,V\otimes U\right). $$ Equivalently, for all valid indices $i,k\in\{1,\ldots,r\}$ and $j,\ell\in\{1,\ldots,c\}$, $$ \operatorname{Cov}(X_{ij},X_{k\ell})=U_{ik}V_{j\ell}. $$

The first statement identifies the complete joint law; the second gives the covariance between any two entries directly. The order of the Kronecker factors follows the column-wise stacking convention: $V$ acts on column indices, while each block of the covariance matrix corresponds to a pair of columns and contains row-index covariances from $U$.

Derivation from the Density

Write $Y=X-M$, $y=\operatorname{vec}(Y)$, and use the identity $\operatorname{vec}(AYB)=(B^{T}\otimes A)\operatorname{vec}(Y)$. The trace in the matrix-normal density can then be rewritten as a quadratic form in $y$:

\operatorname{tr}(V^{-1}Y^{T}U^{-1}Y) =y^{T}(V^{-1}\otimes U^{-1})y.

Because $U$ and $V$ are symmetric, the Kronecker matrix $V^{-1}\otimes U^{-1}$ is symmetric positive definite. Its determinant is $|V^{-1}|^{r}|U^{-1}|^{c}$, so its square-root determinant is $|V|^{-r/2}|U|^{-c/2}$. Substituting the quadratic form and determinant into the density gives

f_Y(Y)=(2\pi)^{-rc/2}|V^{-1}\otimes U^{-1}|^{1/2} \exp\!\left\{-\frac12y^{T}(V^{-1}\otimes U^{-1})y\right\}.

This is exactly the density of a zero-mean $rc$-dimensional normal vector with precision matrix $V^{-1}\otimes U^{-1}$. Since $(V^{-1}\otimes U^{-1})^{-1}=V\otimes U$, its covariance is $V\otimes U$. Adding back the mean shifts the vector mean to $\operatorname{vec}(M)$ without changing the covariance. This proves the distributional claim.

For the entry-wise formula, the covariance between positions $i+(j-1)r$ and $k+(\ell-1)r$ in the vector is the corresponding entry of $V\otimes U$. By the definition of the Kronecker product, that entry equals $V_{j\ell}U_{ik}$. Therefore $\operatorname{Cov}(X_{ij},X_{k\ell})=U_{ik}V_{j\ell}$, as stated. $\square$

Worked Examples

Worked Example: Building the Covariance Matrix in Column Order

Suppose $X$ is $2\times2$, with row scale and column scale

U=\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix}, \qquad V=\begin{pmatrix}3&1\\1&2\end{pmatrix}.

Both matrices are positive definite: their leading diagonal entries are positive, and their determinants are $2(1)-0.5^2=1.75>0$ and $3(2)-1^2=5>0$, respectively. The column-wise vector is $(X_{11},X_{21},X_{12},X_{22})^T$. Its covariance is

V\otimes U = \begin{pmatrix}3U&1U\\1U&2U\end{pmatrix} = \begin{pmatrix} 6&1.5&2&0.5\\ 1.5&3&0.5&1\\ 2&0.5&4&1\\ 0.5&1&1&2 \end{pmatrix}.

For a direct check, the covariance between the second and third vector entries is $0.5$. These entries are $X_{21}$ and $X_{12}$, so the entry-wise formula gives $U_{21}V_{12}=0.5(1)=0.5$. The diagonal entry for $X_{22}$, at position four, is $U_{22}V_{22}=1(2)=2$, matching the displayed matrix.

Worked Example: The Same Matrix Stacked Row by Row

Use the same $U$ and $V$, but now list the entries of $X$ row by row: $(X_{11},X_{12},X_{21},X_{22})^T$. This changes the ordering of the four coordinates, so the covariance matrix must be permuted as well. In this order, the covariance is $U\otimes V$:

U\otimes V = \begin{pmatrix}2V&0.5V\\0.5V&V\end{pmatrix} = \begin{pmatrix} 6&2&1.5&0.5\\ 2&4&0.5&1\\ 1.5&0.5&3&1\\ 0.5&1&1&2 \end{pmatrix}.

For example, positions two and three now represent $X_{12}$ and $X_{21}$. Their covariance is $U_{12}V_{12}=0.5(1)=0.5$, which appears at row two, column three. The order of the factors has not changed the underlying pairwise covariances; it has changed how those covariances are placed in the vector covariance matrix.

Worked Example: Checking a Block and a Cross-Covariance

Let $X$ be $2\times3$, with

U=\begin{pmatrix}2&-1\\-1&3\end{pmatrix}, \qquad V=\begin{pmatrix}1&0&0\\0&2&0\\0&0&4\end{pmatrix}.

The row scale is positive definite because its first diagonal entry is $2>0$ and its determinant is $2(3)-(-1)^2=5>0$; the diagonal column scale has positive diagonal entries. Column-wise stacking gives $\operatorname{vec}(X)=(X_{11},X_{21},X_{12},X_{22},X_{13},X_{23})^T$. Because $V$ is diagonal, the covariance matrix consists of three diagonal blocks:

V\otimes U= \begin{pmatrix} 2&-1&0&0&0&0\\ -1&3&0&0&0&0\\ 0&0&4&-2&0&0\\ 0&0&-2&6&0&0\\ 0&0&0&0&8&-4\\ 0&0&0&0&-4&12 \end{pmatrix}.

The variance of $X_{22}$, the fourth entry, is $U_{22}V_{22}=3(2)=6$. The covariance of $X_{22}$ and $X_{12}$, entries four and three, is $U_{21}V_{22}=(-1)(2)=-2$, as shown in the matrix. Entries from different columns have zero covariance here because the off-diagonal entries of $V$ are zero.

Common Mistake: Reversing the Kronecker Factors

A frequent error is to write $U\otimes V$ while using the column-wise vector $\operatorname{vec}(X)$. That expression has the right overall dimension, so the mistake can be difficult to notice from dimensions alone. But its blocks correspond to a different coordinate order. In column-wise order, the entry at vector position $i+(j-1)r$ is $X_{ij}$, and its covariance with $X_{k\ell}$ must be $U_{ik}V_{j\ell}$. This index check gives $V\otimes U$, not $U\otimes V$.

A related error is to treat the scale matrices as if they were already the covariance matrix of the vectorized observation. They are not: the full covariance has $rc$ rows and columns, and its entries combine one row-scale entry with one column-scale entry. The two scale matrices also have different dimensions in general, which can expose a reversed-factor error, but dimensions alone do not settle the issue when $r=c$. Always state the stacking order and check at least one covariance entry by its indices.

Why the Derivation Is Useful

Vectorization connects a matrix-normal model to the ordinary multivariate normal distribution. Once the order is fixed, its mean, covariance, and density can be read in standard vector form. This is useful whenever a calculation is expressed in terms of matrix entries but the available normal-theory tools are stated for vectors. The derivation also gives a practical verification rule: any proposed covariance matrix must reproduce $\operatorname{Cov}(X_{ij},X_{k\ell})=U_{ik}V_{j\ell}$ at the positions assigned by the chosen vectorization.

The essential convention is concise: column-wise vectorization gives $V\otimes U$, with column scale first and row scale second. If a different stacking order is used, the covariance coordinates must be reordered consistently.

Check Your Understanding

Use column-wise vectorization unless a question specifies otherwise.

For an $r\times c$ matrix, at what position in $\operatorname{vec}(X)$ does $X_{ij}$ appear?
If $X\sim MN_{r,c}(M,U,V)$, what are the mean and covariance of $\operatorname{vec}(X)$?
For the entry-wise covariance formula, which indices of $U$ and $V$ correspond to $\operatorname{Cov}(X_{ij},X_{k\ell})$?
Why does row-wise stacking lead to $U\otimes V$ rather than the column-wise formula?
For a $2\times2$ matrix with $U_{21}=-0.4$ and $V_{12}=1.5$, calculate $\operatorname{Cov}(X_{21},X_{12})$.