Gaussian Likelihood for the Common-Factor Model

Why the Gaussian Likelihood Has Two Covariance Terms

A common-factor model proposes a structured covariance matrix for a multivariate observation. Maximum likelihood assesses that proposal by asking how well the resulting multivariate normal distribution accounts for the observed data. Two features of the covariance matter: its determinant, which measures overall volume, and a trace term, which measures how the observed scatter sits relative to the proposed covariance.

The goal here is to derive that criterion and interpret its terms. The criterion is useful even when the optimization over loadings and uniquenesses must be carried out numerically. It also makes clear why minimizing residual correlations alone is not the same as maximizing a Gaussian likelihood.

The strict positivity of the uniquenesses ensures that \(\Sigma\) is positive definite: for every nonzero \(a\in\mathbb{R}^p\), \(a^{\mathsf T}\Sigma a=\|\Lambda^{\mathsf T}a\|^2+\sum_{j=1}^p\psi_j a_j^2>0\). Consequently its inverse and determinant are well-defined, with \(\det(\Sigma)>0\). This tutorial concerns the likelihood for this observed-data covariance; it does not treat the latent factors as observed.

Deriving the Covariance Log-Likelihood

Suppose \(x_1,\ldots,x_n\) are independent observations from \(N_p(\mu,\Sigma)\). For now, take \(\mu\) as specified. The multivariate normal density for one observation is

Independence makes the joint likelihood the product of these densities. Taking logarithms turns the product into a sum. Define the scatter matrix about \(\mu\) by \(S_\mu=n^{-1}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^{\mathsf T}\). The resulting expression is the central likelihood formula.

Proof. The log of the product of the \(n\) densities is

For a vector \(y\), the scalar \(y^{\mathsf T}\Sigma^{-1}y\) equals \(\operatorname{tr}(\Sigma^{-1}yy^{\mathsf T})\). Applying this identity to each summand and using linearity of trace gives

Substitution proves the formula. The factor-model version follows because its covariance is positive definite under the stated condition on \(\Psi\). \(\square\)

When the mean is unknown and estimated freely, define \(\bar{x}=n^{-1}\sum_i x_i\) and \(S=n^{-1}\sum_i(x_i-\bar{x})(x_i-\bar{x})^{\mathsf T}\). For any proposed \(\mu\), the scatter decomposes as

To verify this, write \(x_i-\mu=(x_i-\bar{x})+(\bar{x}-\mu)\), expand the outer product, and sum. The cross terms vanish because \(\sum_i(x_i-\bar{x})=0\). Thus the log-likelihood can also be written

For fixed positive-definite \(\Sigma\), the final quadratic form is nonnegative. It is a nonnegative contribution to the negative log-likelihood, or equivalently a nonpositive contribution to the log-likelihood. It is minimized at \(\mu=\bar{x}\), so the likelihood over an unrestricted mean is maximized there.

Worked Examples: Reading the Likelihood Criterion

What the Criterion Does—and Does Not—Say

For a fixed data set, the Gaussian likelihood depends on the observations through the mean and scatter matrices appearing in the formula. Once the mean is set to \(\bar{x}\), its covariance-dependent part is completely described by \(S\), \(\log\det(\Sigma)\), and \(\operatorname{tr}(S\Sigma^{-1})\). The parameters \(\Lambda\) and \(\Psi\) matter through the covariance they produce; different parameter values that produce the same \(\Sigma\) have the same Gaussian likelihood.

The determinant term penalizes covariance matrices with large volume, while the trace term evaluates the observed scatter after it is measured in the geometry induced by \(\Sigma^{-1}\). Neither term should be judged in isolation: a candidate with a smaller determinant need not have a smaller total criterion. The factor structure constrains the covariance candidates, so maximizing likelihood means finding the best fit within that structured set—not choosing an arbitrary positive-definite matrix.

A common mistake is to maximize the likelihood by minimizing only \(\operatorname{tr}(S\Sigma^{-1})\). That ignores \(\log\det(\Sigma)\); inflating a covariance can reduce the trace while increasing the determinant penalty. The correction is to compare the full sum. Another sign error is to describe the mean-offset quadratic as increasing the log-likelihood. It is nonnegative in the negative log-likelihood and therefore lowers the log-likelihood unless the offset is zero.

These formulas presume an admissible positive-definite covariance. If a uniqueness is allowed to reach zero, positive definiteness is no longer guaranteed by the stated condition, and the inverse and log-determinant require separate treatment. In ordinary interior likelihood calculations, keeping every \(\psi_j>0\) makes the criterion well-defined.