Why Gaussian Likelihood Calculations Go Wrong
Gaussian likelihoods have a compact form, but several terms that look like harmless bookkeeping can affect the answer. A missing factor of \(n\) changes a covariance estimate; a determinant treated as a constant can change the optimizer; and an estimate found without a constraint may not solve the constrained problem. In likelihood comparisons, dropping a term is safe only when it is genuinely common to the models being compared.
Let \(X_1,\ldots,X_n\) be independent \(p\)-dimensional observations with \(X_i\sim N_p(\mu,\Sigma)\), where \(\mu\in\mathbb{R}^p\) and \(\Sigma\) is positive definite. Write the observed vectors as \(x_1,\ldots,x_n\), and define \(\bar{x}=n^{-1}\sum_{i=1}^n x_i\). The log-likelihood is
The determinant is that of the covariance matrix itself, not of a standard-deviation matrix. Its logarithm is multiplied by \(n/2\), while the sum of quadratic forms is multiplied by \(1/2\). These coefficients are linked: the quadratic term sums over \(n\) observations, and the determinant term accounts for the normalizing factor in each observation's density.
Covariance Normalization: Maximum Likelihood Is Not Unbiased Estimation
For an unrestricted mean, the Gaussian maximum-likelihood estimate is \(\hat{\mu}=\bar{x}\). Define the residual scatter matrix \(A=\sum_{i=1}^n(x_i-\bar{x})(x_i-\bar{x})^\mathsf{T}\). When \(A\) is positive definite, maximizing the likelihood over positive-definite covariance matrices gives \(\hat{\Sigma}=A/n\). The divisor is \(n\), not \(n-1\). The familiar divisor \(n-1\) produces an unbiased estimator under the usual independent Gaussian sampling assumptions, but it is not the maximum-likelihood estimate.
Proof. For fixed \(\mu\), terms not depending on \(\Sigma\) can be omitted when maximizing, leaving
Set \(B=A/n\), which is positive definite, and write \(\Sigma=B^{1/2}CB^{1/2}\) for a positive-definite matrix \(C\). Then \(\log|\Sigma|=\log|B|+\log|C|\), and \(\operatorname{tr}(\Sigma^{-1}A)=n\operatorname{tr}(C^{-1})\). Thus maximizing the likelihood over \(C\) is equivalent to minimizing \(\log|C|+\operatorname{tr}(C^{-1})\). If the positive eigenvalues of \(C\) are \(\lambda_1,\ldots,\lambda_p\), this quantity is \(\sum_{j=1}^p(\log\lambda_j+1/\lambda_j)\). The derivative of each summand is \((\lambda_j-1)/\lambda_j^2\), so the summand decreases up to \(1\) and increases after \(1\). Its unique minimum is at \(1\). Therefore \(C=I\) and \(\Sigma=B=A/n\). For an unrestricted mean, the quadratic sum decomposes as
For every positive-definite \(\Sigma\), the second term contributes a nonnegative amount to the quadratic form, minimized at \(\mu=\bar{x}\). The fixed-mean result then gives the stated covariance estimate. \(\square\)
Worked Example: Dividing by \(n\) or \(n-1\)
Take the scalar observations \(1,2,4\), so \(n=3\). Their mean is \(\bar{x}=7/3\), and the sum of squared deviations is
The maximum-likelihood variance estimate is therefore \((14/3)/3=14/9\). The unbiased sample variance is \((14/3)/2=7/3\). They are different estimates for different purposes. Substituting \(7/3\) into a calculation that specifically requires the Gaussian maximum-likelihood estimate changes the fitted likelihood; calling it “the Gaussian MLE” is incorrect.
Determinant Terms Must Match the Quadratic Term
A common coefficient error is to include \(-\tfrac12\log|\Sigma|\) only once for the whole sample. In fact, each of the \(n\) Gaussian densities contributes \(-\tfrac12\log|\Sigma|\), giving \(-\tfrac n2\log|\Sigma|\). The error matters because the determinant term penalizes the size of the covariance matrix, while the quadratic term rewards covariance matrices that fit the observed residuals.
Worked Example: A Missing Factor of \(n\)
Consider four two-dimensional observations \((2,0),(-2,0),(0,\sqrt{2}),(0,-\sqrt{2})\), with the mean estimated freely. Their sample mean is \((0,0)\), and their residual scatter is \(A=\operatorname{diag}(8,4)\). The correct covariance estimate is
To see what the coefficient error does, suppose the determinant term is mistakenly written \(-\tfrac12\log|\Sigma|\), but the quadratic term is still summed over all four observations. The erroneous objective, up to terms independent of \(\Sigma\), is
This has the same form as the correct fixed-mean likelihood with one determinant contribution and scatter \(A\); its maximizer is \(\Sigma=A=\operatorname{diag}(8,4)\). The erroneous answer is four times the correct covariance estimate. The problem is not a harmless change in notation: the estimate changes because the determinant coefficient no longer matches the four quadratic contributions.
There is a related error in which \(\log|\Sigma|\) is dropped entirely on the grounds that it is “just a determinant term.” If \(\Sigma\) is being estimated, this term depends on the parameter. In many settings, omitting it destroys the trade-off that produces a finite maximum. For instance, a quadratic term alone can be made smaller by inflating covariance in directions containing residuals, while the determinant term penalizes that inflation.
Parameter Constraints Must Be Imposed During Optimization
When a parameter is restricted, the likelihood must be maximized over the restricted parameter space. Estimating without the constraint and then adjusting the resulting estimate afterward generally does not solve the constrained optimization problem. The effect is especially clear when the covariance couples coordinates: changing one mean coordinate can change the best-fitting value of another.
Worked Example: A Constrained Mean with Correlated Coordinates
Suppose two observations both equal \((2,0)^\mathsf{T}\), and the covariance is known to be
Restrict the mean to satisfy \(\mu_1=0\), leaving \(\mu_2\) free. Since the sample mean is \((2,0)^\mathsf{T}\), minimizing the negative log-likelihood is equivalent to minimizing \((2,-\mu_2)\Sigma^{-1}(2,-\mu_2)^\mathsf{T}\). Differentiating with respect to \(\mu_2\) gives the condition that the second coordinate of \(\Sigma^{-1}(2,-\mu_2)^\mathsf{T}\) is zero:
Thus the constrained maximum-likelihood estimate is \((0,-1.6)^\mathsf{T}\), not the post-hoc adjustment \((0,0)^\mathsf{T}\) obtained by simply setting the first coordinate of the unrestricted estimate \((2,0)^\mathsf{T}\) to zero. The quadratic form per observation at the constrained optimum is \((2,1.6)\Sigma^{-1}(2,1.6)^\mathsf{T}=4\); for two observations it is \(8\). At the post-hoc value \((0,0)^\mathsf{T}\), the quadratic form per observation is \(4/0.36=100/9\), or \(200/9\) for the two observations. The post-hoc value is not only a different estimate; using it would also give a different likelihood comparison.
Which Likelihood Constants Can Be Dropped?
A term may be omitted while maximizing over a parameter if it does not depend on that parameter. In particular, the Gaussian term \(-np\log(2\pi)/2\) is constant with respect to \(\mu\) and \(\Sigma\) for fixed \(n\) and \(p\). It also cancels from a likelihood ratio comparing models for the same observed data and dimension. But omitting it means the displayed quantity is a log-likelihood only up to an additive constant; it is not the absolute log-likelihood.
Proof. Subtracting the two log-likelihoods gives \(\ell_1(y)-\ell_0(y)=g_1(y)-g_0(y)\), since \(c(y)-c(y)=0\). Exponentiating both sides gives \(\exp(\ell_1(y)-\ell_0(y))=\exp(g_1(y)-g_0(y))\), so the likelihood ratio is unchanged as well. This argument requires the same term in both models; it does not justify dropping a term that depends on model parameters or differs between the compared likelihoods. \(\square\)
Worked Example: A Gaussian Constant Cancels in a Likelihood Ratio
For two observations \(0,2\) from \(N(0,\sigma^2)\), compare the fixed-variance models \(\sigma^2=1\) and \(\sigma^2=4\). The full log-likelihood is \(-\log(2\pi\sigma^2)-\tfrac12(0^2+2^2)/\sigma^2\). Therefore
The common term \(-\log(2\pi)\) cancels, giving \(\ell(1)-\ell(4)=\log 4-3/2\). The likelihood-ratio statistic for comparing variance \(1\) with variance \(4\), written as \(2\{\ell(4)-\ell(1)\}\), is \(3-2\log 4\), approximately \(0.227\). Omitting the common constant does not change this calculation. By contrast, the term \(-\log(\sigma^2)\) is not common across these models and must be retained.
A Practical Audit
Before accepting a Gaussian likelihood calculation, check the model and the algebra in a fixed order. This helps distinguish a notational shortcut from an error that changes the result.
For \(n\) independent \(p\)-vectors, check for \(-np\log(2\pi)/2\), \(-n\log|\Sigma|/2\), and the summed quadratic form with coefficient \(-1/2\).
Record which means and covariance entries are free, fixed, or constrained before differentiating or optimizing.
Do not substitute an unbiased covariance estimate for a maximum-likelihood estimate without explicitly changing the calculation.
For an estimate, a term must be parameter-independent to be discarded. For a model comparison, it must be common to both likelihoods.
A covariance parameter must be positive definite. If the residual scatter is singular, the interior covariance maximum described above does not apply; the likelihood may fail to have a finite maximum over positive-definite covariance matrices.
Common Mistakes
- Using \(n-1\) and calling the result the MLE. The divisor \(n-1\) gives the usual unbiased sample covariance; the Gaussian covariance MLE uses \(n\) when the mean is estimated freely.
- Writing one determinant term for the whole sample. Each observation contributes a Gaussian normalizing factor, so the log-determinant coefficient is \(-n/2\).
- Calling a parameter-dependent term a constant. If \(\Sigma\) is being optimized, \(\log|\Sigma|\) varies with the parameter and cannot be discarded.
- Imposing a restriction after estimation. The constrained estimate must optimize the likelihood within the constrained space; coordinate-wise adjustment can fail when the covariance couples coordinates.
- Dropping constants inconsistently. A common additive term cancels in a likelihood ratio only when it is identical for both models on the same data.
Check Your Understanding
Use the likelihood structure and checks in this tutorial to answer the following questions.
- For independent \(p\)-dimensional Gaussian observations, what coefficient multiplies \(\log|\Sigma|\) in the full log-likelihood, and why does it involve \(n\)?
- For the scalar observations \(1,2,4\), explain why \(14/9\) and \(7/3\) are both recognizable variance estimates but only one is the maximum-likelihood estimate.
- If the residual scatter matrix is \(A\) and the mean is fixed, what covariance estimate maximizes the Gaussian likelihood when \(A\) is positive definite?
- Why can an additive constant be omitted from a likelihood ratio for two models fitted to the same data, but not automatically from a comparison involving different data?
- In the constrained-mean example, why does setting the first coordinate of the unrestricted estimate to zero fail to produce the constrained maximum?