Concentrating the Gaussian Likelihood over Model Parameters

From Gaussian Data to a Covariance Objective

In Gaussian factor analysis, the observations are modeled through a covariance matrix assembled from loadings and specific variances. Maximum likelihood therefore searches over parameters, but the data enter the covariance part of the objective through a compact summary: the sample covariance matrix. Deriving this reduction makes the optimization target precise and clarifies which constraints the factor model imposes.

Assume \(X_1,\ldots,X_n\) are independent observations from a \(p\)-variate normal distribution with mean vector \(\mu\) and covariance matrix \(\Sigma\). For a common-factor model with \(q\) factors, take the factor covariance to be the identity matrix and assume the specific errors are uncorrelated with one another and with the factors. The implied covariance is \(\Sigma=\Lambda\Lambda^{\mathsf T}+\Psi\), where \(\Lambda\) is a \(p\times q\) loading matrix and \(\Psi\) is diagonal.

The strict positivity condition on each uniqueness \(\psi_j\) guarantees that \(\Sigma\) is positive definite, even when the loading matrix does not have full column rank. Indeed, for every nonzero vector \(a\), \(a^{\mathsf T}\Sigma a=\|\Lambda^{\mathsf T}a\|^2+\sum_j\psi_j a_j^2>0\). Thus its determinant and inverse, both needed in the normal likelihood, are well defined.

Profiling Out the Mean

For fixed positive-definite \(\Sigma\), the Gaussian log-likelihood for the full sample is

The mean can be maximized out before optimizing the factor parameters. The following derivation identifies its maximizing value and shows exactly where \(S\) enters.

To prove the result, write \(X_i-\mu=(X_i-\bar X)+(\bar X-\mu)\) and expand the quadratic sum. The cross terms add to zero because \(\sum_i(X_i-\bar X)=0\). Therefore

The second term is nonnegative and equals zero exactly when \(\mu=\bar X\), since \(\Sigma^{-1}\) is positive definite. For the first term, use \(\operatorname{tr}(uv^{\mathsf T}A)=v^{\mathsf T}Au\) to obtain

Substitution into the log-likelihood proves both claims. The result does not require \(S\) to be invertible: the trace expression remains defined because \(\Sigma\) is positive definite. \(\square\)

The Constrained Factor-Analysis Optimization

Since the leading term of \(\ell_{\mathrm{prof}}\) is constant in the parameters, maximizing the profile likelihood is equivalent to minimizing

over \(\Lambda\in\mathbb R^{p\times q}\) and diagonal \(\Psi\) with strictly positive entries. This is the concentrated Gaussian likelihood objective. Its two terms have distinct roles: the log determinant penalizes covariance volume, while the trace term measures how the model covariance weights the observed sample covariance. Neither term alone is the likelihood criterion; both arise from the normal density.

For a proof, let \(A=S^{-1/2}\Sigma S^{-1/2}\), which is positive definite, and let \(\lambda_1,\ldots,\lambda_p\) be its positive eigenvalues. Determinant multiplication and cyclic invariance of trace give

For every \(\lambda>0\), \(h(\lambda)=\log\lambda+1/\lambda-1\) is nonnegative. In fact, \(h'(\lambda)=(\lambda-1)/\lambda^2\), so it decreases up to \(1\), increases after \(1\), and has minimum \(h(1)=0\). Summing this inequality over the eigenvalues proves the stated bound. Equality requires every \(\lambda_j=1\), which means \(A=I\) and hence \(\Sigma=S\). Conversely, \(\Sigma=S\) gives equality directly. \(\square\)

This result supplies a useful benchmark, not an automatic solution to factor analysis. The unconstrained covariance optimum is \(S\) when \(S\) is positive definite. The factor model attains that benchmark only if \(S\) can be represented as \(\Lambda\Lambda^{\mathsf T}+\Psi\) with the chosen factor count and positive diagonal uniquenesses. Otherwise the maximum-likelihood fit is the best covariance within the restricted model, not an arbitrary covariance matrix.

What the Concentrated Likelihood Does—and Does Not—Say

For fixed data and a fixed factor count, the profiled likelihood depends on \(\Lambda\) and \(\Psi\) only through their implied covariance matrix. Thus two parameter values that produce the same \(\Sigma\) have the same likelihood. The likelihood itself does not distinguish parameter representations that leave the covariance unchanged; constraints used to select or identify a representation are separate from the covariance objective.

The Gaussian assumption is essential to this reduction: under a multivariate normal model, the covariance parameters enter the likelihood through the log determinant and the quadratic form summarized by \(S\). This should not be generalized to an arbitrary non-Gaussian model merely because its covariance is specified by the same factor decomposition.

Common Mistake

A tempting but incorrect argument is: “The sample covariance is the maximum-likelihood covariance, so the factor-analysis estimate is \(S\).” The first part is true only for an unrestricted positive-definite covariance model when \(S\) is positive definite. In factor analysis, the covariance must also have the specified loading-plus-diagonal form. If \(S\) does not have that form for the chosen \(q\), it is not an admissible factor-model covariance. The correction is to minimize \(F(\Lambda,\Psi)\) over the stated constraints; the saturated benchmark tells us the best possible value, not that the restricted model must attain it.