Worked Matrix-Normal Density And Probability Calculations

From the Matrix Model to Numerical Answers

A matrix-normal model is useful only when its parameters can be translated into quantities of interest. A density calculation requires careful handling of two determinants and a trace; a covariance question requires tracking both a row index and a column index. For probabilities, a linear summary of the random matrix is itself normally distributed, but its variance depends on both scale matrices.

The calculations below use small matrices so each step can be checked directly. Vectorization supplies an independent check: stacking the columns of a matrix-normal random matrix gives an ordinary multivariate normal vector. This is particularly useful for catching a swapped row/column scale or an incorrect covariance.

Definition: Let \(X\) be an \(r\times c\) random matrix with mean matrix \(M\), where \(U\) is an \(r\times r\) positive-definite row scale matrix and \(V\) is a \(c\times c\) positive-definite column scale matrix. We write \(X\sim MN_{r,c}(M,U,V)\) when its density is

f(X)=(2\pi)^{-rc/2}|U|^{-c/2}|V|^{-r/2} \exp\!\left\{-\frac12\operatorname{tr}\!\left[U^{-1}(X-M)V^{-1}(X-M)^T\right]\right\}.

Here density is with respect to the product of the \(rc\) entries of \(X\). With column-wise vectorization, \(\operatorname{vec}(X)\sim N_{rc}(\operatorname{vec}(M),V\otimes U)\).

The row scale \(U\) appears next to the row dimension, and the column scale \(V\) appears next to the column dimension. In particular, the entrywise covariance is \(\operatorname{Cov}(X_{ij},X_{k\ell})=U_{ik}V_{j\ell}\). These conventions will be used throughout the calculations.

Evaluating a Matrix-Normal Density

A reliable density calculation has three separate parts: compute the determinants, form the centered matrix \(D=X-M\), and evaluate the trace in the exponent. The determinant factor is a scalar; the trace is also a scalar, even though its argument is a matrix.

Worked Example: Density at a Specified Matrix

Suppose \(X\sim MN_{2,2}(M,U,V)\), with \(M=\begin{pmatrix}0&0\\0&0\end{pmatrix}\), \(U=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), and \(V=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}\). Evaluate the density at \(X=\begin{pmatrix}1&0\\0&1\end{pmatrix}\).

The determinants and inverses are \(|U|=4-1=3\), \(|V|=1-1/4=3/4\), \(U^{-1}=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}\), and \(V^{-1}=\frac43\begin{pmatrix}1&-1/2\\-1/2&1\end{pmatrix}\). The centered matrix is \(D=X-M=I_2\).

Because \(D=I_2\), the trace term is \(\operatorname{tr}(U^{-1}V^{-1})\). Multiplying,

U^{-1}V^{-1} =\begin{pmatrix}2/3&-1/3\\-1/3&2/3\end{pmatrix} \begin{pmatrix}4/3&-2/3\\-2/3&4/3\end{pmatrix} =\begin{pmatrix}10/9&-8/9\\-8/9&10/9\end{pmatrix}, \qquad \operatorname{tr}(U^{-1}V^{-1})=\frac{20}{9}.

Since \(r=c=2\), the determinant part of the density is \((2\pi)^{-2}|U|^{-1}|V|^{-1} =(4\pi^2)^{-1}(1/3)(4/3)=1/(9\pi^2)\). The exponent is \(-\frac12(20/9)=-10/9\). Therefore,

f(X)=\frac{1}{9\pi^2}\exp\!\left(-\frac{10}{9}\right).

This is a density value, not the probability that \(X\) equals this one matrix. For a continuously distributed random matrix, an individual point has probability zero; probabilities concern regions of the \(2\times2\) matrix space.

Checking the Density by Vectorization

For column-wise vectorization, the matrix \(\begin{pmatrix}x_{11}&x_{12}\\x_{21}&x_{22}\end{pmatrix}\) becomes \((x_{11},x_{21},x_{12},x_{22})^T\). In the preceding example, the covariance of this four-dimensional vector is \(V\otimes U\). Its determinant must agree with the determinant factors in the matrix density.

Result: For positive-definite \(U\) and \(V\), the vectorized normal density and the matrix-normal density give the same determinant factor and quadratic form. In particular, \(|V\otimes U|=|V|^r|U|^c\), and the vectorized quadratic form is \(\operatorname{vec}(D)^T(V^{-1}\otimes U^{-1})\operatorname{vec}(D) =\operatorname{tr}(U^{-1}DV^{-1}D^T)\), where \(D=X-M\).

To verify the determinant identity, diagonalize the positive-definite matrices as \(U=Q\Lambda Q^T\) and \(V=R\Gamma R^T\), with eigenvalues \(\lambda_1,\ldots,\lambda_r\) and \(\gamma_1,\ldots,\gamma_c\). The eigenvalues of \(V\otimes U\) are all \(c r\) products \(\gamma_j\lambda_i\). Their product is \(\prod_{j=1}^c\prod_{i=1}^r\gamma_j\lambda_i =(\prod_j\gamma_j)^r(\prod_i\lambda_i)^c=|V|^r|U|^c\). Taking the inverse square root of this determinant gives \(|V|^{-r/2}|U|^{-c/2}\), exactly the matrix-density factor.

For the quadratic-form identity, use \(\operatorname{vec}(D)^T(A\otimes B)\operatorname{vec}(D) =\operatorname{tr}(D^TBD A^T)\). Taking \(A=V^{-1}\) and \(B=U^{-1}\), both symmetric, gives \(\operatorname{tr}(D^TU^{-1}DV^{-1})\). The trace is unchanged by cyclically moving the first factor to the end, so this equals \(\operatorname{tr}(U^{-1}DV^{-1}D^T)\). Thus both forms agree. \(\square\)

For the numerical density example, \(\operatorname{vec}(D)=(1,0,0,1)^T\). The diagonal entries of \(V^{-1}\otimes U^{-1}\) corresponding to its first and fourth coordinates are each \(8/9\), and the corresponding off-diagonal entry is \(2/9\). Hence the quadratic form is \(8/9+8/9+2(2/9)=20/9\), matching the trace calculation.

Entrywise Covariances

The covariance formula makes the two sources of dependence explicit. The row scale determines how entries in different rows co-vary, and the column scale determines how entries in different columns co-vary. Both factors matter: the covariance of \(X_{ij}\) and \(X_{k\ell}\) is the product of the corresponding row and column entries.

Result: If \(X\sim MN_{r,c}(M,U,V)\), then \(\operatorname{Cov}(X_{ij},X_{k\ell})=U_{ik}V_{j\ell}\). This follows because the coordinates of \(\operatorname{vec}(X)\), in column-wise order, have covariance matrix \(V\otimes U\): the covariance between the coordinate for \(X_{ij}\) and that for \(X_{k\ell}\) is precisely the corresponding Kronecker-product entry.

Worked Example: Covariances Among Four Entries

Let \(X\sim MN_{2,2}(M,U,V)\), where \(U=\begin{pmatrix}2&1\\1&2\end{pmatrix}\) and \(V=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}\). Find the variances of the entries and the covariance between \(X_{11}\) and \(X_{22}\).

For each entry, its variance is \(U_{ii}V_{jj}\). Thus \(\operatorname{Var}(X_{11})=2(1)=2\), \(\operatorname{Var}(X_{21})=2(1)=2\), \(\operatorname{Var}(X_{12})=2(1)=2\), and \(\operatorname{Var}(X_{22})=2(1)=2\). For the two diagonal entries, both indices change, so \(\operatorname{Cov}(X_{11},X_{22})=U_{12}V_{12}=1(1/2)=1/2\).

The vectorized covariance matrix makes the same checks visible:

\operatorname{Cov}(\operatorname{vec}X) =V\otimes U =\begin{pmatrix} 2&1&1&1/2\\ 1&2&1/2&1\\ 1&1/2&2&1\\ 1/2&1&1&2 \end{pmatrix}.

The first and fourth coordinates represent \(X_{11}\) and \(X_{22}\), respectively. Their covariance is the \((1,4)\) entry, \(1/2\), agreeing with the entrywise formula. The diagonal entries are all 2, also agreeing with the four variances just computed.

Probabilities for Linear Summaries

A linear summary can be written as \(a^T X b\), where \(a\) is an \(r\)-vector and \(b\) is a \(c\)-vector. Its vectorized form is \((b\otimes a)^T\operatorname{vec}(X)\). A linear transformation of a multivariate normal vector is normal, so this summary is univariate normal.

Result: If \(X\sim MN_{r,c}(M,U,V)\), \(a\in\mathbb R^r\), and \(b\in\mathbb R^c\), then \(a^TXb\sim N(a^TMb,(a^TUa)(b^TVb))\). Indeed, its mean is \(a^TMb\), and its variance, using the vectorized covariance, is \((b\otimes a)^T(V\otimes U)(b\otimes a) =(b^TVb)(a^TUa)\).

Worked Example: A Tail Probability for a Matrix Summary

Suppose \(X\sim MN_{2,2}(M,U,V)\), with \(M=\begin{pmatrix}1&0\\0&1\end{pmatrix}\), \(U=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), and \(V=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}\). Find \(P(X_{11}+X_{22}>2+\sqrt{5})\).

Choose \(a=b=(1,1)^T\). Then \(a^TXb=X_{11}+X_{12}+X_{21}+X_{22}\), which is not the desired summary. Instead, use the coefficient matrix \(C=I_2\) and the summary \(\operatorname{tr}(C^TX)=X_{11}+X_{22}\). In vectorized form this is \(c^T\operatorname{vec}(X)\), with \(c=(1,0,0,1)^T\).

The mean of the summary is \(\operatorname{tr}(M)=2\). Its variance can be computed entrywise:

\operatorname{Var}(X_{11}+X_{22}) =\operatorname{Var}(X_{11})+\operatorname{Var}(X_{22}) +2\operatorname{Cov}(X_{11},X_{22}) =2+2+2(1/2)=5.

The vectorized check uses the first and fourth coordinates of \(V\otimes U\). Each has variance 2, and their covariance is \(1/2\), so \(c^T(V\otimes U)c=2+2+2(1/2)=5\) as well. Therefore \(S=X_{11}+X_{22}\sim N(2,5)\). Standardizing the requested event gives

P(S>2+\sqrt{5}) =P\!\left(\frac{S-2}{\sqrt{5}}>1\right) =1-\Phi(1)\approx 0.1587,

where \(\Phi\) is the standard normal cumulative distribution function. The covariance term is essential: omitting it would produce variance 4 and a different tail probability.

Common Mistakes and Practical Checks

The matrix-normal density and probability calculations are compact, but compact notation can hide errors. A short set of checks helps locate them before they affect a final answer.

Swapping the row and column scales. Under the convention here, \(\operatorname{Cov}(\operatorname{vec}X)=V\otimes U\), not \(U\otimes V\). Check the vectorization order and the matrix dimensions.
Dropping one determinant factor. The density uses \(|U|^{-c/2}|V|^{-r/2}\). The exponent on each determinant depends on the other matrix dimension.
Using a density as a point probability. A density value describes local concentration per unit volume; it is not \(P(X=x)\).
Ignoring covariance in a sum. For two entries, \(\operatorname{Var}(A+B)=\operatorname{Var}(A)+\operatorname{Var}(B)+2\operatorname{Cov}(A,B)\). Entries in different rows and columns need not be independent.
Applying \(a^TXb\) to the wrong summary. This form gives a rank-one set of weights \(a_i b_j\). A general weighted sum \(\operatorname{tr}(C^TX)\) instead has vectorized coefficient \(\operatorname{vec}(C)\); use that coefficient directly when the weights do not factor as \(a_i b_j\).

For any proposed calculation, compare the matrix expression with its vectorized equivalent. Agreement between the trace and quadratic form, or between an entrywise variance and a Kronecker-product entry, is a strong arithmetic and notation check.

Check Your Understanding

Use the matrix-normal density and vectorization conventions stated above.

For an \(r\times c\) matrix-normal variable, what powers of \(|U|\) and \(|V|\) appear in the density, and why do their exponents involve the opposite dimensions?
If \(U_{ik}=0\), what is \(\operatorname{Cov}(X_{ij},X_{k\ell})\) for any \(j,\ell\)? Explain using the entrywise covariance formula.
For column-wise vectorization of a \(2\times2\) matrix, which vector coordinates correspond to \(X_{11}\) and \(X_{22}\)?
For a summary \(a^TXb\), state its mean and variance in terms of \(M,U,V,a,\) and \(b\).
Why is \(f(X)\) at one specified matrix not the probability that the random matrix equals that matrix?