Maximum Likelihood Under Diagonal Gaussian Covariance

Why a Diagonal Covariance Model Separates

When measurements on several variables are modeled jointly, the covariance matrix describes both their individual variability and their dependence. A diagonal covariance model makes a particular restriction: different coordinates have zero covariance. For a multivariate normal distribution, that restriction also means the coordinates are independent. This structure makes the likelihood simpler: each coordinate contributes a separate one-dimensional Gaussian likelihood, and each coordinate’s mean and variance can be estimated from its own observations.

We derive the estimates and state the condition under which they are genuine maxima. That condition matters: if the observed values of any coordinate are all identical, its variance can shrink toward zero and the likelihood has no finite maximum.

Model and Likelihood

Suppose \(X_1,\ldots,X_n\) are independent observations from a \(p\)-dimensional normal distribution with mean vector \(\mu=(\mu_1,\ldots,\mu_p)^\mathsf{T}\) and diagonal covariance matrix \(\Sigma\). Write \(X_i=(X_{i1},\ldots,X_{ip})^\mathsf{T}\), and let \(\Sigma=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_p^2)\), where every \(\sigma_j^2>0\). The observed value of \(X_{ij}\) is denoted by \(x_{ij}\).

The density of one observation is the product of its \(p\) coordinate densities. Independence across observations then gives the full likelihood:

The brackets are important: each coordinate’s variance factor and exponential term belong to the same coordinate-specific factor. Equivalently, collecting all exponential terms gives

Taking logarithms turns the product into a sum:

Each summand involves only the parameters \(\mu_j,\sigma_j^2\) and the data in coordinate \(j\). Thus maximizing the whole likelihood is equivalent to maximizing each coordinate’s two-parameter likelihood separately. This follows directly from the multivariate normal density: its determinant term is \(|\Sigma|=\prod_j\sigma_j^2\), and its quadratic form is \(\sum_j(x_{ij}-\mu_j)^2/\sigma_j^2\). Both split into coordinate contributions, as does the likelihood across independent observations.

Deriving the Maximum-Likelihood Estimates

For a fixed coordinate \(j\), set \(Q_j(\mu_j)=\sum_{i=1}^n(x_{ij}-\mu_j)^2\). The identity \(Q_j(\mu_j)=S_j+n(\mu_j-\bar{x}_j)^2\) follows by writing \(x_{ij}-\mu_j=(x_{ij}-\bar{x}_j)+(\bar{x}_j-\mu_j)\) and using \(\sum_i(x_{ij}-\bar{x}_j)=0\). Therefore \(Q_j\) is minimized at \(\mu_j=\bar{x}_j\), with minimum \(S_j\).

Proof. For fixed \(\sigma_j^2>0\), the coordinate log-likelihood is maximized by minimizing \(Q_j(\mu_j)\). The decomposition above shows that this unique minimizing mean is \(\bar{x}_j\). If \(S_j>0\), substituting that mean leaves, apart from a constant, the function of \(v=\sigma_j^2>0\)

Its derivative is \(g_j'(v)=(S_j-nv)/(2v^2)\), which is positive for \(0<v<S_j/n\) and negative for \(v>S_j/n\). Also, \(g_j(v)\to-\infty\) as \(v\downarrow0\) because \(S_j>0\), and \(g_j(v)\to-\infty\) as \(v\to\infty\). Hence its unique global maximum is at \(v=S_j/n\). Since the full log-likelihood is a sum of these coordinate log-likelihoods, the coordinatewise maxima give the unique joint maximum when every \(S_j>0\).

If \(S_j=0\), all observed values in coordinate \(j\) equal \(\bar{x}_j\). Set \(\mu_j=\bar{x}_j\); the residual sum of squares is then zero, and its log-likelihood contribution is \(-\tfrac n2\log v\). This tends to \(+\infty\) as \(v\downarrow0\). The other coordinates can be held at fixed admissible parameter values, so the full likelihood is unbounded above. Since \(v=0\) is not in the model, no finite maximum is attained. \(\square\)

The estimates are therefore

when all coordinates vary in the sample. The variance estimate uses denominator \(n\), not \(n-1\): maximum likelihood optimizes the likelihood, whereas the familiar \(n-1\) denominator gives an unbiased estimate of a univariate variance under the usual sampling assumptions.

Worked Examples

Interpretation and Common Pitfalls

The diagonal restriction does more than set estimated off-diagonal covariances to zero after fitting an unrestricted model. It defines a restricted likelihood in which coordinate \(j\)'s parameters are estimated from coordinate \(j\)'s observations alone. The fitted covariance matrix is diagonal by construction; the data do not supply estimates of cross-coordinate covariance parameters in this model.

• Putting an exponential outside the coordinate product. The likelihood must include one exponential contribution for every coordinate. Writing a product only over the variance factors while leaving an exponential with an unbound coordinate index does not define the intended likelihood.
• Using \(n-1\) for the maximum-likelihood variance. The estimate is \(S_j/n\). Replacing \(n\) by \(n-1\) changes the estimator and does not maximize this likelihood.
• Assuming every data set has a finite variance MLE. If any coordinate is constant across the observations, its likelihood increases without bound as its fitted variance approaches zero.
• Confusing zero covariance with unrestricted dependence. Under a multivariate normal model, diagonal covariance implies independent coordinates. That conclusion depends on normality; zero covariance alone does not generally imply independence.