Matrix-Normal Representation of Gaussian Samples

From Gaussian Observations to a Random Matrix

A sample of multivariate observations can be stored as a matrix: each row represents one observation, and each column represents one variable. This arrangement is more than a convenient data format. When the rows are independent Gaussian vectors with a common mean and covariance, the whole matrix has a matrix-normal distribution. Its covariance separates into a part describing relationships among rows and a part describing relationships among columns.

Throughout, let \(X\) be an \(n\) by \(p\) random matrix. We use column-wise vectorization: \(\operatorname{vec}(X)\) stacks the first column of \(X\), then the second, and so on. This choice fixes the order of the Kronecker products in the covariance formula.

Definition: Suppose \(U\) is an \(n\) by \(n\) positive-definite matrix and \(V\) is a \(p\) by \(p\) positive-definite matrix. The matrix \(X\) has a matrix-normal distribution, written \(X\sim\mathcal{MN}_{n,p}(M,U,V)\), if \(\operatorname{vec}(X)\) has a multivariate normal distribution with mean \(\operatorname{vec}(M)\) and covariance \(V\otimes U\). Here \(M\) is an \(n\) by \(p\) mean matrix, \(U\) is the row covariance matrix, and \(V\) is the column covariance matrix.

The dimensions indicate the roles of the two covariance matrices. \(U\) describes covariance between rows for a fixed pair of columns; \(V\) describes covariance between columns for a fixed pair of rows. In particular, for row indices \(i,i'\) and column indices \(j,j'\),

\operatorname{Cov}(X_{ij},X_{i'j'})=U_{ii'}V_{jj'}.

This product form is called a separable covariance structure. It is a restriction: an arbitrary covariance matrix for the \(np\) entries need not be expressible as \(V\otimes U\).

Independent Gaussian Observations as a Special Case

Suppose \(X_1,\ldots,X_n\) are independent \(p\)-dimensional random vectors with the same distribution \(N_p(\mu,\Sigma)\). Put their transposes into the rows of \(X\), so that row \(i\) is \(X_i^\mathsf{T}\). The mean matrix has every row equal to \(\mu^\mathsf{T}\). Independence gives identity row covariance, and the covariance among variables within each row is \(\Sigma\).

M=\boldsymbol{1}_n\mu^\mathsf{T}, \qquad X\sim\mathcal{MN}_{n,p}(M,I_n,\Sigma), \qquad \operatorname{Cov}(\operatorname{vec}(X))=\Sigma\otimes I_n.

The Kronecker product here is \(\Sigma\otimes I_n\), not \(I_n\otimes\Sigma\), because vectorization stacks columns. Its block structure places an \(n\) by \(n\) block \(\Sigma_{jj'}I_n\) at block position \((j,j')\). Thus entries in different rows of the same column are uncorrelated, while different columns in the same row can be correlated.

Worked Example: Three Independent Bivariate Observations

Let \(X_1,X_2,X_3\) be independent with distribution \(N_2(\mu,\Sigma)\), where \(\mu=(1,-1)^\mathsf{T}\) and \(\Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}\). Arrange them as rows of a \(3\) by \(2\) matrix \(X\). Identify its matrix-normal parameters and calculate selected covariances.

The mean matrix and row covariance are \(\displaystyle M=\begin{pmatrix}1&-1\\1&-1\\1&-1\end{pmatrix}\) and \(U=I_3\). The column covariance is \(V=\Sigma\). This matrix is positive definite because its first diagonal entry is positive and its determinant is \(4(9)-1(1)=35>0\).

With column-wise vectorization, the entries are ordered as \(\operatorname{vec}(X)=(X_{11},X_{21},X_{31},X_{12},X_{22},X_{32})^\mathsf{T}\). Therefore, \(\operatorname{Cov}(\operatorname{vec}(X))=\Sigma\otimes I_3\), or explicitly, \(\displaystyle\begin{pmatrix}4I_3&I_3\\I_3&9I_3\end{pmatrix}\).

For example, \(\operatorname{Var}(X_{21})=U_{22}V_{11}=1\cdot4=4\), and \(\operatorname{Cov}(X_{21},X_{22})=U_{22}V_{12}=1\cdot1=1\). Across rows, \(\operatorname{Cov}(X_{11},X_{21})=U_{12}V_{11}=0\cdot4=0\). These calculations match independence of the observations and the within-observation covariance \(\Sigma\).

Entrywise Covariance and the Vectorized Form

The defining Kronecker covariance has a direct interpretation in terms of individual matrix entries. The following result derives that interpretation and makes clear which matrix controls each kind of covariance.

Theorem: If \(X\sim\mathcal{MN}_{n,p}(M,U,V)\), then \(E[X_{ij}]=M_{ij}\) and \(\operatorname{Cov}(X_{ij},X_{i'j'})=U_{ii'}V_{jj'}\) for every pair of entries. In particular, when \(U=I_n\), distinct rows are independent because the matrix is jointly Gaussian and their cross-covariances are zero.

Proof. Since \(X\) is jointly Gaussian by definition, write \(X=M+U^{1/2}ZV^{1/2}\), where \(Z\) has \(np\) independent \(N(0,1)\) entries and the square roots are the symmetric positive-definite square roots. Set \(A=U^{1/2}\) and \(B=V^{1/2}\). Then \(X_{ij}=M_{ij}+\sum_{k=1}^n\sum_{\ell=1}^p A_{ik}Z_{k\ell}B_{j\ell}\). The zero means of the entries of \(Z\) give \(E[X_{ij}]=M_{ij}\). Independence and unit variances of those entries give

\begin{aligned} \operatorname{Cov}(X_{ij},X_{i'j'}) &=\sum_{k,\ell}\sum_{r,s} A_{ik}A_{i'r}B_{j\ell}B_{j's} \operatorname{Cov}(Z_{k\ell},Z_{rs})\\ &=\sum_{k,\ell}A_{ik}A_{i'k}B_{j\ell}B_{j'\ell}\\ &=(AA^\mathsf{T})_{ii'}(BB^\mathsf{T})_{jj'}\\ &=U_{ii'}V_{jj'}. \end{aligned}

For \(U=I_n\), the covariance between entries in two distinct rows is zero for every pair of columns. Each row is a subvector of the jointly Gaussian vector \(\operatorname{vec}(X)\); Gaussian subvectors with zero cross-covariance are independent. Hence the rows are independent. \(\square\)

Worked Example: Reading Row and Column Covariance Separately

Let \(X\sim\mathcal{MN}_{2,2}(0,U,V)\), where \(\displaystyle U=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\) and \(\displaystyle V=\begin{pmatrix}4&1\\1&9\end{pmatrix}\). Find the covariance matrix of \(\operatorname{vec}(X)\) and two covariances between entries.

Both matrices are positive definite: \(\det(U)=1-0.25=0.75>0\), and \(\det(V)=36-1=35>0\), with positive diagonal entries. Column-wise vectorization gives \(\operatorname{Cov}(\operatorname{vec}(X))=V\otimes U\). Multiplying the blocks yields

V\otimes U = \begin{pmatrix} 4U&U\\ U&9U \end{pmatrix} = \begin{pmatrix} 4&2&1&0.5\\ 2&4&0.5&1\\ 1&0.5&9&4.5\\ 0.5&1&4.5&9 \end{pmatrix}.

The vector order is \((X_{11},X_{21},X_{12},X_{22})^\mathsf{T}\). The covariance between entries in different rows and different columns is \(\operatorname{Cov}(X_{11},X_{22})=U_{12}V_{12}=0.5\cdot1=0.5\), the fourth entry in the first row of the displayed covariance matrix. Meanwhile, \(\operatorname{Cov}(X_{11},X_{21})=U_{12}V_{11}=0.5\cdot4=2\). The first calculation uses both row and column covariance; the second holds the column fixed and uses only the row covariance multiplier.

The Matrix-Normal Density

When \(U\) and \(V\) are positive definite, the matrix-normal distribution has a density with respect to Lebesgue measure on the \(np\) entries. Its formula follows from transforming a matrix of independent standard normal entries.

Theorem: If \(U\) and \(V\) are positive definite and \(X\sim\mathcal{MN}_{n,p}(M,U,V)\), then

f_X(X)= \frac{\exp\!\left[-\frac12\operatorname{tr}\!\left\{ V^{-1}(X-M)^\mathsf{T}U^{-1}(X-M) \right\}\right]} {(2\pi)^{np/2}|U|^{p/2}|V|^{n/2}}.

Derivation. Let \(A=U^{1/2}\), \(B=V^{1/2}\), and write \(X=M+AZB^\mathsf{T}\), where \(Z\) has independent standard normal entries. Its density is \((2\pi)^{-np/2}\exp\{-\operatorname{tr}(Z^\mathsf{T}Z)/2\}\). Under column-wise vectorization, \(\operatorname{vec}(X-M)=(B\otimes A)\operatorname{vec}(Z)\). The absolute Jacobian determinant of this linear transformation is \(|B\otimes A|=|B|^p|A|^n=|V|^{n/2}|U|^{p/2}\).

The inverse transformation is \(Z=A^{-1}(X-M)B^{-\mathsf{T}}\). Substituting it into the exponent and using invariance of trace under cyclic rearrangement gives \(\operatorname{tr}(Z^\mathsf{T}Z)=\operatorname{tr}\{V^{-1}(X-M)^\mathsf{T}U^{-1}(X-M)\}\). The change-of-variables formula therefore gives the stated density, including its normalizing constant. \(\square\)

Worked Example: Simplifying the Density for Independent Rows

Suppose \(X\sim\mathcal{MN}_{n,p}(\boldsymbol{1}_n\mu^\mathsf{T},I_n,\Sigma)\), with \(\Sigma\) positive definite. Show how the matrix-normal density reduces to the product of the \(n\) row densities.

Because \(|I_n|=1\), the density formula becomes \(\displaystyle (2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\{-\tfrac12\operatorname{tr}[\Sigma^{-1}(X-M)^\mathsf{T}(X-M)]\}\). Write row \(i\) of \(X\) as \(X_i^\mathsf{T}\). The matrix \(X-M\) has rows \((X_i-\mu)^\mathsf{T}\), so \((X-M)^\mathsf{T}(X-M)=\sum_{i=1}^n(X_i-\mu)(X_i-\mu)^\mathsf{T}\). Linearity of trace gives

\operatorname{tr}\{\Sigma^{-1}(X-M)^\mathsf{T}(X-M)\} =\sum_{i=1}^n (X_i-\mu)^\mathsf{T}\Sigma^{-1}(X_i-\mu).

Also, \((2\pi)^{-np/2}|\Sigma|^{-n/2}\) is the product of \(n\) copies of \((2\pi)^{-p/2}|\Sigma|^{-1/2}\). Thus the joint density is exactly the product of the \(N_p(\mu,\Sigma)\) densities, as required for independent rows.

Constructing Covariance Through Row and Column Transformations

A useful way to build matrix-normal random matrices is to transform a matrix \(Z\) of independent standard normal entries. For fixed matrices \(A\) and \(B\), the matrix \(AZB^\mathsf{T}\) has row covariance \(AA^\mathsf{T}\) and column covariance \(BB^\mathsf{T}\), provided these matrices are positive definite. This construction can represent correlated rows as well as correlated columns.

Worked Example: Transforming Independent Standard Normal Entries

Let \(Z\) be a \(2\) by \(2\) matrix of independent \(N(0,1)\) entries, and set \(\displaystyle A=\begin{pmatrix}1&0\\1&1\end{pmatrix}\), \(\displaystyle B=\begin{pmatrix}1&0\\0&2\end{pmatrix}\), and \(X=AZB^\mathsf{T}\). Find the row and column covariance matrices and verify two entrywise covariances.

Multiplication gives \(\displaystyle X=\begin{pmatrix}Z_{11}&2Z_{12}\\Z_{11}+Z_{21}&2Z_{12}+2Z_{22}\end{pmatrix}\). The covariance factors are \(\displaystyle U=AA^\mathsf{T}=\begin{pmatrix}1&1\\1&2\end{pmatrix}\) and \(\displaystyle V=BB^\mathsf{T}=\begin{pmatrix}1&0\\0&4\end{pmatrix}\). They are positive definite: \(\det(U)=1\) and \(\det(V)=4\), and their diagonal entries are positive. Hence \(X\sim\mathcal{MN}_{2,2}(0,U,V)\).

Directly from the displayed entries, \(\operatorname{Var}(X_{21})=\operatorname{Var}(Z_{11}+Z_{21})=1+1=2\), agreeing with \(U_{22}V_{11}=2\cdot1=2\). Also, \(\operatorname{Cov}(X_{11},X_{21})=\operatorname{Cov}(Z_{11},Z_{11}+Z_{21})=1\), agreeing with \(U_{12}V_{11}=1\cdot1=1\). In contrast, \(\operatorname{Cov}(X_{11},X_{12})=0\), since the corresponding standard normal entries are independent; the formula gives \(U_{11}V_{12}=1\cdot0=0\).

Why the Representation Matters

The matrix-normal form keeps two sources of dependence distinct. The row covariance describes dependence among observations, while the column covariance describes dependence among variables. For independent Gaussian observations with a common covariance, the row factor is \(I_n\); introducing a nonidentity row factor instead allows dependence across observations. This separation can simplify notation and calculations when the covariance genuinely has the product form.

The restriction to a Kronecker product matters. Specifying \(U\) and \(V\) does not describe every possible covariance pattern among \(np\) entries. It asserts that the covariance of two entries factors into a row contribution and a column contribution. A covariance model that does not satisfy this structure cannot be represented by a single matrix-normal distribution with those dimensions.

Pitfall: reversing the Kronecker factors. With column-wise vectorization, the covariance is \(V\otimes U\). Writing \(U\otimes V\) generally gives a differently ordered covariance matrix.
Pitfall: treating \(U\) as covariance among variables. \(U\) indexes the rows; \(V\) indexes the columns. In an independent sample with variables in columns, use \(U=I_n\) and \(V=\Sigma\).
Pitfall: assuming zero covariance means independence without checking the distribution. Zero covariance alone does not imply independence in general. Here the conclusion for disjoint rows follows because the joint distribution is Gaussian.

Key takeaway: For column-wise vectorization, \(X\sim\mathcal{MN}_{n,p}(M,U,V)\) means \(\operatorname{vec}(X)\) has mean \(\operatorname{vec}(M)\) and covariance \(V\otimes U\). Its entrywise covariance is \(U_{ii'}V_{jj'}\); independent Gaussian observations are the special case \(U=I_n\).

Check Your Understanding

Answer each question by identifying the relevant matrix dimensions and covariance factors.

If \(n\) independent rows each have distribution \(N_p(\mu,\Sigma)\), what are \(M\), \(U\), and \(V\) in the matrix-normal representation?
For column-wise vectorization, which Kronecker product gives \(\operatorname{Cov}(\operatorname{vec}(X))\), and why does its order matter?
If \(U_{12}=0.4\) and \(V_{23}=3\), what is \(\operatorname{Cov}(X_{12},X_{23})\)?
What additional condition, beyond zero covariance between two rows, justifies concluding that they are independent?
In \(X=AZB^\mathsf{T}\) with independent standard normal entries in \(Z\), what are the row and column covariance matrices?