Kronecker Covariance for Multidimensional Arrays

For two dimensions this becomes \(\operatorname{Cov}\{\operatorname{vec}(X)\}=\Sigma_2\otimes\Sigma_1\), not \(\Sigma_1\otimes\Sigma_2\). The Kronecker product \(A\otimes B\) is the block matrix whose \((i,j)\) block is \(A_{ij}B\). Thus the entry of \(\Sigma_d\otimes\cdots\otimes\Sigma_1\) corresponding to two array cells is the product of the factor entries for their respective indices.

Justification. Consider two cells \(X_{i_1,\ldots,i_d}\) and \(X_{j_1,\ldots,j_d}\). By the definition of the Kronecker product, the matrix entry in row indexed by \((i_d,\ldots,i_1)\) and column indexed by \((j_d,\ldots,j_1)\) of \(\Sigma_d\otimes\cdots\otimes\Sigma_1\) is $$ (\Sigma_d)_{i_dj_d}\cdots(\Sigma_1)_{i_1j_1} = \prod_{k=1}^{d}(\Sigma_k)_{i_kj_k}. $$ The vectorization convention assigns those row and column positions to the corresponding array cells. The resulting matrix therefore has exactly the covariance specified by the separable formula. \(\square\)

What the Factors Say About Dependence

For any two cells, the covariance is the product of the factor entries at the corresponding pairs of indices. If two cells agree in one coordinate, the contribution from that dimension is a diagonal factor entry. If factor matrices are scaled to have unit diagonal, they are correlation matrices, and the covariance between the cells is the product of their factor correlations. In that case, the variance of every cell is \(1\), and the covariance formula is also a correlation formula.

More generally, normalizing the covariance between two cells by their standard deviations gives $$ \operatorname{Corr}(X_{i_1,\ldots,i_d},X_{j_1,\ldots,j_d}) = \prod_{k=1}^{d} \frac{(\Sigma_k)_{i_kj_k}} {\sqrt{(\Sigma_k)_{i_ki_k}(\Sigma_k)_{j_kj_k}}}. $$ This follows because the variance of a cell is \(\prod_k(\Sigma_k)_{i_ki_k}\), so the square root of the product of the two cell variances separates into the corresponding factorwise square roots. Thus the model says not only that covariance separates, but also that correlations across dimensions multiply.

Positive Definiteness of the Full Covariance

A covariance matrix must be positive semidefinite. In this model, taking every factor to be positive definite guarantees the full Kronecker covariance is positive definite as well.

Proof. For each \(k\), the spectral theorem gives an orthogonal matrix \(Q_k\) and a diagonal matrix \(\Lambda_k\) with positive diagonal entries such that \(\Sigma_k=Q_k\Lambda_kQ_k^\mathsf{T}\). The Kronecker product rules give $$ \Sigma_d\otimes\cdots\otimes\Sigma_1 = (Q_d\otimes\cdots\otimes Q_1) (\Lambda_d\otimes\cdots\otimes\Lambda_1) (Q_d\otimes\cdots\otimes Q_1)^\mathsf{T}. $$ The first factor is orthogonal. The middle matrix is diagonal, with diagonal entries equal to products of one positive eigenvalue from each \(\Lambda_k\). Every such product is positive. For any nonzero vector \(v\), an orthogonal change of coordinates preserves nonzeroness, so the quadratic form of the displayed matrix at \(v\) is a sum of positive diagonal entries times squared coordinates, with at least one nonzero squared coordinate. It is strictly positive. The matrix is therefore positive definite. \(\square\)

Counting the Factor Parameters

A symmetric \(m_k\times m_k\) factor has \(m_k\) diagonal entries and \(m_k(m_k-1)/2\) distinct off-diagonal entries. Its nominal number of free entries is therefore $$ \frac{m_k(m_k+1)}{2}. $$ With \(d\) unrestricted symmetric factors, the nominal factor-parameter count is $$ \sum_{k=1}^{d}\frac{m_k(m_k+1)}{2}. $$ By comparison, the covariance matrix of the full vectorized array has dimension \(M\times M\), where \(M=\prod_{k=1}^{d}m_k\), and an unrestricted symmetric covariance matrix has \(M(M+1)/2\) free entries. Positive definiteness restricts the allowed values of these parameters but does not change these dimensions of the parameter spaces. The factor count is a count of factor coordinates; it should not automatically be interpreted as a count of distinct covariance matrices without checking whether different factor choices can represent the same covariance.

Common Mistake and Model Interpretation

A common mistake is to reverse the factor order or to treat a covariance in the vectorized array as a single factor entry. Under column-stacking for a matrix, the correct order is \(B\otimes A\) when \(A\) describes rows and \(B\) describes columns. The entry for two cells is \(A_{i i'}B_{j j'}\), not just \(A_{i i'}\) or \(B_{j j'}\). Checking one pair of cells directly against the array-index formula is a reliable way to catch a mismatched ordering.

A second pitfall is to infer that a Kronecker model is appropriate simply because it uses fewer parameters. It imposes a substantive restriction: the covariance between cells must factor into dimension-specific contributions. If dependence along one dimension changes with the level of another dimension, that separability may be inadequate. The model matters because it can make covariance estimation and interpretation more manageable, but its structural assumption should be assessed in the context of the measurements.