Block-Diagonal Covariance Models and Likelihood

Why Block-Diagonal Covariance?

Measurements often come in groups: several laboratory assays, repeated outcomes from related domains, or collections of variables recorded by different instruments. A block-diagonal covariance model expresses a specific simplification: measurements in different groups have zero covariance, while dependence within each group remains unrestricted. For Gaussian data, this covariance structure has a stronger implication—independence between groups—and makes likelihood estimation separable.

The partition must be specified in advance for the model described here. The goal is not to discover which variables belong together, but to understand the statistical consequences of imposing a given partition and to derive its Gaussian maximum-likelihood estimates.

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed \(p\)-dimensional observations from \(N_p(\mu,\Sigma)\), with unrestricted mean vector \(\mu\) and covariance \(\Sigma\) in this model. Write \(X_{ig}\) and \(\mu_g\) for the coordinates of observation \(i\) and the mean vector in group \(g\). Define the group sample mean and within-group scatter matrix by \( \bar X_g=n^{-1}\sum_{i=1}^n X_{ig} \) and \( C_g=\sum_{i=1}^n (X_{ig}-\bar X_g)(X_{ig}-\bar X_g)^\mathsf{T} \). The corresponding sample covariance with divisor \(n\) is \(C_g/n\).

What the Structure Says About Independence

For a multivariate normal vector, zero covariance between two subvectors implies their independence. Here is the density argument. Because \(\Sigma\) is block diagonal, its determinant is the product of block determinants and its inverse is block diagonal with blocks \(\Sigma_g^{-1}\). Therefore the Gaussian density at \(x=(x_1^\mathsf{T},\ldots,x_G^\mathsf{T})^\mathsf{T}\) is

Each factor is the marginal density for group \(g\), so the joint density factors into the product of the group densities. That factorization proves mutual independence of the groups. This implication uses joint normality: for a general distribution, zero cross-group covariances do not by themselves establish independence.

Gaussian Likelihood and the Blockwise Estimates

For a candidate mean \(\mu\) and block covariance matrices \(\Sigma_1,\ldots,\Sigma_G\), the log-likelihood is

The group-specific terms involve only the observations, mean, and covariance for that group. The next result gives the MLE when each within-group scatter matrix is positive definite.

Proof. First fix the covariance matrices and consider the mean. For each group, expanding around \(\bar X_g\) gives

After multiplying by \(\Sigma_g^{-1}\) and taking traces, the second term contributes \(n(\bar X_g-\mu_g)^\mathsf{T}\Sigma_g^{-1}(\bar X_g-\mu_g)\), which is nonnegative and is zero only when \(\mu_g=\bar X_g\). Thus the likelihood for fixed covariance is uniquely maximized at the group sample mean.

Substitute these means. The part of the log-likelihood depending on \(\Sigma_g\) is \( -\frac n2\{\log|\Sigma_g|+\operatorname{tr}(\Sigma_g^{-1}C_g/n)\} \). Because \(-n/2\) is a negative factor, maximizing this expression is equivalent to minimizing the expression in braces. Put \(A=C_g/n\), which is positive definite by hypothesis, and write \(\Sigma_g=A^{1/2}BA^{1/2}\), with \(B\) positive definite. Determinant multiplication and the cyclic property of trace give

Let \(\lambda_1,\ldots,\lambda_{p_g}\) be the positive eigenvalues of \(B\). The terms depending on \(B\) sum to \(\sum_j(\log\lambda_j+1/\lambda_j)\). For \(\lambda>0\), the derivative of \(\log\lambda+1/\lambda\) is \((\lambda-1)/\lambda^2\); hence this function decreases up to \(1\), increases after \(1\), and has its unique minimum at \(1\). The sum is therefore uniquely minimized when every \(\lambda_j=1\), which forces \(B=I\) and \(\Sigma_g=A=C_g/n\). This holds separately for every group and proves the result. \(\square\)

Existence, Parameter Count, and a Singular-Scatter Pitfall

The positive-definiteness condition in the theorem matters. If a within-group scatter matrix is singular, the likelihood over positive-definite block covariances has no finite maximum: it can increase without bound as the covariance shrinks along a direction with zero observed scatter.

To verify the singular case, set the mean to the group sample mean and diagonalize a singular \(C_g\) in an orthonormal basis. Suppose its rank is \(r<p_g\), with positive eigenvalues \(c_1,\ldots,c_r\) and \(p_g-r\) zero eigenvalues. In this basis choose \(\Sigma_g\) to have eigenvalues \(c_j/n\) in the positive-scatter directions and \(\varepsilon\) in the zero-scatter directions. For every \(\varepsilon>0\), this covariance is positive definite. The trace \(\operatorname{tr}(\Sigma_g^{-1}C_g)\) is \(nr\), independent of \(\varepsilon\), while \( \log|\Sigma_g|=\sum_{j=1}^r\log(c_j/n)+(p_g-r)\log\varepsilon \). As \(\varepsilon\) decreases to zero, the log-determinant term in the log-likelihood tends to \(+\infty\), since its coefficient is \(-n/2\). Other groups can be held fixed, so the full likelihood is unbounded above. This proves the result.

For unknown means, \(C_g\) has rank at most \(\min(p_g,n-1)\), because the centered observations sum to zero. Thus \(n\ge p_g+1\) is necessary for positive definiteness, though it does not guarantee it for every possible data set. In continuously distributed, nondegenerate Gaussian samples, it is positive definite with probability one when \(n\ge p_g+1\).

The model also reduces the number of free covariance parameters. A symmetric unrestricted \(p\times p\) covariance matrix has \(p(p+1)/2\) distinct entries. Under the specified block structure, the count is \(\sum_{g=1}^G p_g(p_g+1)/2\); all cross-block entries are fixed at zero, not estimated. This reduction can stabilize estimation and simplify interpretation, but it is justified only when the imposed zero cross-group covariances are appropriate.

Common Mistakes

• Equating zero covariance with independence for every distribution. The independence conclusion here follows from the Gaussian density factorization. Outside the Gaussian setting, zero covariance alone is insufficient.
• Dividing by a negative likelihood factor without reversing the optimization. The covariance objective is minimized because the log-likelihood contains it multiplied by \(-n/2\), a negative number.
• Using a singular sample covariance as an ordinary MLE. If any within-block scatter is singular, there is no finite maximizer over positive-definite covariances; the likelihood is unbounded.
• Confusing the divisor \(n\) with \(n-1\). The Gaussian maximum-likelihood covariance estimate is \(C_g/n\), not the unbiased sample covariance \(C_g/(n-1)\).
• Assuming the model removes all within-group association. Covariances inside each block remain unrestricted; only covariances between distinct blocks are fixed to zero.