Matrix Log-Likelihood for Unknown Mean and Covariance

From Individual Observations to a Matrix Expression

For multivariate normal data, the likelihood begins as a product of densities, one for each observed vector. That form is useful for stating the model, but it hides an important structure: the data enter the log-likelihood through a sample mean and a scatter matrix. Rewriting the expression makes it possible to distinguish how a candidate mean and a candidate covariance affect the fit.

Let $\mathbf{x}_1,\ldots,\mathbf{x}_n$ be independent observations in $\mathbb{R}^p$, each with distribution $N_p(\boldsymbol{\mu},\boldsymbol{\Sigma})$. Here $\boldsymbol{\mu}\in\mathbb{R}^p$ is the candidate mean vector, and $\boldsymbol{\Sigma}$ is a candidate $p\times p$ symmetric positive-definite covariance matrix. Positive definiteness ensures that the multivariate normal density exists, that $|\boldsymbol{\Sigma}|>0$, and that $\boldsymbol{\Sigma}^{-1}$ is defined.

Definition: The sample mean is $\bar{\mathbf{x}}=n^{-1}\sum_{i=1}^n\mathbf{x}_i$. The scatter matrix about the sample mean is $\mathbf{C}=\sum_{i=1}^n(\mathbf{x}_i-\bar{\mathbf{x}})(\mathbf{x}_i-\bar{\mathbf{x}})^\mathsf{T}$. It is a sum of outer products, with no division by $n$ or $n-1$.

The scatter matrix is symmetric and positive semidefinite. It need not be invertible: for example, it is singular if the centered observations do not span all of $\mathbb{R}^p$. That causes no problem in the likelihood expression below, which requires the inverse of the candidate covariance, not the inverse of the scatter matrix.

Definition: For observed data $\mathbf{x}_1,\ldots,\mathbf{x}_n$, the likelihood $L(\boldsymbol{\mu},\boldsymbol{\Sigma})$ is their joint density viewed as a function of the candidate parameters. The log-likelihood is $\ell(\boldsymbol{\mu},\boldsymbol{\Sigma})=\log L(\boldsymbol{\mu},\boldsymbol{\Sigma})$.

For one observation, the multivariate normal density is $f(\mathbf{x}_i)=(2\pi)^{-p/2}|\boldsymbol{\Sigma}|^{-1/2}\exp\{-\tfrac12(\mathbf{x}_i-\boldsymbol{\mu})^\mathsf{T}\boldsymbol{\Sigma}^{-1}(\mathbf{x}_i-\boldsymbol{\mu})\}$. Independence makes the joint likelihood the product of these $n$ densities. Taking its logarithm gives

\ell(\boldsymbol{\mu},\boldsymbol{\Sigma}) =-\frac{np}{2}\log(2\pi) -\frac{n}{2}\log|\boldsymbol{\Sigma}| -\frac12\sum_{i=1}^n (\mathbf{x}_i-\boldsymbol{\mu})^\mathsf{T} \boldsymbol{\Sigma}^{-1} (\mathbf{x}_i-\boldsymbol{\mu}).

The first term is constant with respect to both candidate parameters. The determinant term depends on the covariance, and the final term measures the observations’ squared deviations from the candidate mean, using the covariance to set the scale and orientation of that distance. The next identity separates the sample’s overall location from its within-sample scatter.

The Sample-Mean and Scatter Decomposition

For any candidate mean $\boldsymbol{\mu}$, write each residual as $\mathbf{x}_i-\boldsymbol{\mu}=(\mathbf{x}_i-\bar{\mathbf{x}})+(\bar{\mathbf{x}}-\boldsymbol{\mu})$. The centered residuals sum to zero. This eliminates the cross-term when the squared residuals are added.

Theorem: For every symmetric matrix $\mathbf{A}$ and every $\boldsymbol{\mu}\in\mathbb{R}^p$, $$ \sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu})^\mathsf{T}\mathbf{A}(\mathbf{x}_i-\boldsymbol{\mu}) =\operatorname{tr}(\mathbf{A}\mathbf{C}) +n(\bar{\mathbf{x}}-\boldsymbol{\mu})^\mathsf{T} \mathbf{A}(\bar{\mathbf{x}}-\boldsymbol{\mu}). $$

To prove the identity, set $\mathbf{d}_i=\mathbf{x}_i-\bar{\mathbf{x}}$ and $\boldsymbol{\delta}=\bar{\mathbf{x}}-\boldsymbol{\mu}$. Then $\sum_i\mathbf{d}_i=\mathbf{0}$, and expanding each quadratic form gives

\begin{aligned} \sum_{i=1}^n(\mathbf{d}_i+\boldsymbol{\delta})^\mathsf{T} \mathbf{A}(\mathbf{d}_i+\boldsymbol{\delta}) &=\sum_{i=1}^n\mathbf{d}_i^\mathsf{T}\mathbf{A}\mathbf{d}_i +2\left(\sum_{i=1}^n\mathbf{d}_i\right)^\mathsf{T}\mathbf{A}\boldsymbol{\delta} +n\boldsymbol{\delta}^\mathsf{T}\mathbf{A}\boldsymbol{\delta}\\ &=\sum_{i=1}^n\mathbf{d}_i^\mathsf{T}\mathbf{A}\mathbf{d}_i +n\boldsymbol{\delta}^\mathsf{T}\mathbf{A}\boldsymbol{\delta}. \end{aligned}

For any vector $\mathbf{d}$, the scalar $\mathbf{d}^\mathsf{T}\mathbf{A}\mathbf{d}$ equals $\operatorname{tr}(\mathbf{A}\mathbf{d}\mathbf{d}^\mathsf{T})$. Summing this equality over $i$ gives $\sum_i\mathbf{d}_i^\mathsf{T}\mathbf{A}\mathbf{d}_i=\operatorname{tr}(\mathbf{A}\mathbf{C})$, which proves the stated identity. $\square$

Now take $\mathbf{A}=\boldsymbol{\Sigma}^{-1}$, which is symmetric because $\boldsymbol{\Sigma}$ is symmetric and positive definite. Substitution into the likelihood yields the matrix log-likelihood:

Key formula: For independent $N_p(\boldsymbol{\mu},\boldsymbol{\Sigma})$ observations, $$ \ell(\boldsymbol{\mu},\boldsymbol{\Sigma}) =-\frac{np}{2}\log(2\pi) -\frac{n}{2}\log|\boldsymbol{\Sigma}| -\frac12\operatorname{tr}(\boldsymbol{\Sigma}^{-1}\mathbf{C}) -\frac{n}{2}(\bar{\mathbf{x}}-\boldsymbol{\mu})^\mathsf{T} \boldsymbol{\Sigma}^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}). $$

The last quadratic form is a squared Mahalanobis distance: deviations in directions with greater covariance receive less weight, while deviations in directions with lower covariance receive more. The trace term summarizes the within-sample scatter relative to the candidate covariance. The log-determinant term reflects the volume scale of that covariance. These terms both depend on $\boldsymbol{\Sigma}$; changing the covariance changes not only the determinant but also the weighting of the observed scatter.

Worked Example: Computing the Decomposition

Suppose $n=3$, $p=2$, and the observations are $\mathbf{x}_1=(1,0)^\mathsf{T}$, $\mathbf{x}_2=(2,1)^\mathsf{T}$, and $\mathbf{x}_3=(3,2)^\mathsf{T}$. Evaluate the log-likelihood at candidate parameters $\boldsymbol{\mu}=(2,0)^\mathsf{T}$ and $\boldsymbol{\Sigma}=\operatorname{diag}(1,4)$.

The sample mean is $\bar{\mathbf{x}}=(2,1)^\mathsf{T}$. The centered vectors are $(-1,-1)^\mathsf{T}$, $(0,0)^\mathsf{T}$, and $(1,1)^\mathsf{T}$, so $\mathbf{C}=\begin{pmatrix}2&2\\2&2\end{pmatrix}$. The candidate covariance has inverse $\boldsymbol{\Sigma}^{-1}=\operatorname{diag}(1,\tfrac14)$ and determinant $4$.

The trace contribution is $\operatorname{tr}(\boldsymbol{\Sigma}^{-1}\mathbf{C})=2+\tfrac12=\tfrac52$. The offset between the sample mean and candidate mean is $(0,1)^\mathsf{T}$, so its contribution before multiplication by $n$ is $(0,1)\operatorname{diag}(1,\tfrac14)(0,1)^\mathsf{T}=\tfrac14$. Thus the complete quadratic sum is $\tfrac52+3(\tfrac14)=\tfrac{13}{4}$. The log-likelihood is $\ell=-3\log(2\pi)-\tfrac32\log 4-\tfrac{13}{8}$, approximately $-9.218$.

How Candidate Means and Covariances Change the Likelihood

The decomposition gives a precise comparison of candidate means when the covariance is held fixed. The determinant and scatter terms are then the same for every candidate mean, so only the Mahalanobis-distance term changes.

Result: For fixed positive-definite $\boldsymbol{\Sigma}$ and any two candidate means $\boldsymbol{\mu}_1,\boldsymbol{\mu}_2$, $$ \ell(\boldsymbol{\mu}_1,\boldsymbol{\Sigma}) -\ell(\boldsymbol{\mu}_2,\boldsymbol{\Sigma}) =-\frac{n}{2}\left[ (\bar{\mathbf{x}}-\boldsymbol{\mu}_1)^\mathsf{T}\boldsymbol{\Sigma}^{-1} (\bar{\mathbf{x}}-\boldsymbol{\mu}_1) -(\bar{\mathbf{x}}-\boldsymbol{\mu}_2)^\mathsf{T}\boldsymbol{\Sigma}^{-1} (\bar{\mathbf{x}}-\boldsymbol{\mu}_2) \right]. $$

Indeed, subtract the two decomposed log-likelihoods. The constant, log-determinant, and trace terms cancel because the covariance and data are unchanged. The two remaining terms are exactly the displayed difference. Therefore, among two candidate means under the same covariance, the one with the smaller squared Mahalanobis distance from $\bar{\mathbf{x}}$ has the larger log-likelihood. This comparison does not require the coordinates to be independent.

Worked Example: A Correlated Covariance Changes the Mean Distance

Suppose $n=3$, $\bar{\mathbf{x}}=(1,2)^\mathsf{T}$, and the candidate covariance is $\boldsymbol{\Sigma}=\begin{pmatrix}2&1\\1&2\end{pmatrix}$. This matrix is positive definite because its eigenvalues are $1$ and $3$, and its inverse is $\tfrac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Compare the candidate means $\boldsymbol{\mu}_1=(1,2)^\mathsf{T}$ and $\boldsymbol{\mu}_2=(2,1)^\mathsf{T}$.

For $\boldsymbol{\mu}_1$, the offset is zero, so its squared Mahalanobis distance is $0$. For $\boldsymbol{\mu}_2$, the offset is $(-1,1)^\mathsf{T}$, and \[ (-1,1)\boldsymbol{\Sigma}^{-1}(-1,1)^\mathsf{T} =(-1,1)(-1,1)^\mathsf{T}=2. \] The mean-dependent contribution for $\boldsymbol{\mu}_2$ is therefore $-\tfrac32(2)=-3$, whereas it is zero for $\boldsymbol{\mu}_1$. With the covariance fixed, the log-likelihood at $\boldsymbol{\mu}_1$ exceeds that at $\boldsymbol{\mu}_2$ by $3$.

When the mean is fixed and the covariance changes, both covariance-dependent terms matter. For any two positive-definite candidate covariances $\boldsymbol{\Sigma}_1,\boldsymbol{\Sigma}_2$, define $\mathbf{Q}_{\boldsymbol{\mu}}=\sum_i(\mathbf{x}_i-\boldsymbol{\mu})(\mathbf{x}_i-\boldsymbol{\mu})^\mathsf{T}$. The quadratic sum in the original likelihood equals $\operatorname{tr}(\boldsymbol{\Sigma}^{-1}\mathbf{Q}_{\boldsymbol{\mu}})$, by the same scalar-to-trace identity used above. Subtracting the two log-likelihood expressions gives

\ell(\boldsymbol{\mu},\boldsymbol{\Sigma}_1) -\ell(\boldsymbol{\mu},\boldsymbol{\Sigma}_2) =-\frac{n}{2}\log\frac{|\boldsymbol{\Sigma}_1|}{|\boldsymbol{\Sigma}_2|} -\frac12\operatorname{tr}\left[ (\boldsymbol{\Sigma}_1^{-1}-\boldsymbol{\Sigma}_2^{-1}) \mathbf{Q}_{\boldsymbol{\mu}} \right].

This identity shows why comparing covariances by determinant alone is insufficient: a covariance that changes volume also changes how the observed residuals are weighted. The determinant records overall volume, while the trace term accounts for the alignment of residual variation with the candidate covariance.

Worked Example: Comparing Two Covariance Candidates

Take four observations in two dimensions: $(0,0)^\mathsf{T}$, $(2,0)^\mathsf{T}$, $(0,2)^\mathsf{T}$, and $(2,2)^\mathsf{T}$. Their sample mean is $(1,1)^\mathsf{T}$, and their scatter matrix is $\mathbf{C}=4\mathbf{I}_2$. Hold the candidate mean at the sample mean and compare $\boldsymbol{\Sigma}_1=\mathbf{I}_2$ with $\boldsymbol{\Sigma}_2=2\mathbf{I}_2$.

For $\boldsymbol{\Sigma}_1$, the determinant is $1$ and $\operatorname{tr}(\boldsymbol{\Sigma}_1^{-1}\mathbf{C})=\operatorname{tr}(4\mathbf{I}_2)=8$. For $\boldsymbol{\Sigma}_2$, the determinant is $4$, its inverse is $\tfrac12\mathbf{I}_2$, and $\operatorname{tr}(\boldsymbol{\Sigma}_2^{-1}\mathbf{C})=\operatorname{tr}(2\mathbf{I}_2)=4$. The mean offset is zero in both cases. Consequently, \[ \ell(\bar{\mathbf{x}},\boldsymbol{\Sigma}_2) -\ell(\bar{\mathbf{x}},\boldsymbol{\Sigma}_1) =-\frac42\log 4-\frac12(4-8) =2-2\log 4 \approx -0.773. \] The larger covariance reduces the trace penalty but increases the log-determinant penalty; the displayed value includes both effects.

Common Pitfall and Practical Interpretation

A frequent error is to replace $\mathbf{C}$ by a sample covariance matrix without adjusting its scale. Here $\mathbf{C}$ is the unnormalized sum of centered outer products. If a sample covariance is defined as $\mathbf{C}/(n-1)$, then the trace term must be written as $(n-1)\operatorname{tr}\{\boldsymbol{\Sigma}^{-1}\mathbf{C}/(n-1)\}$; omitting the factor changes the likelihood.

Another mistake is to treat the covariance contribution as only $-\tfrac{n}{2}\log|\boldsymbol{\Sigma}|$. That expression omits the trace penalty, which depends on the data and on the candidate covariance. Conversely, the trace term alone is not the covariance contribution: the log-determinant must also be included. The decomposition is useful precisely because it keeps these components visible while making clear that the candidate mean is measured relative to the sample mean in the geometry determined by the candidate covariance.

Key takeaway: The Gaussian log-likelihood is a constant minus a covariance log-determinant penalty, a trace penalty for within-sample scatter, and a squared Mahalanobis penalty for the candidate mean’s displacement from the sample mean. Keep the scatter convention and both covariance terms explicit.

Check Your Understanding

Use the definitions and identities above to answer these questions.

Why can the scatter matrix be singular without making the Gaussian likelihood undefined?
In the decomposed log-likelihood, which term changes when the candidate mean changes but the covariance is held fixed?
What does the trace term summarize, and why does it depend on the candidate covariance?
If the scatter matrix is defined as $\mathbf{C}/(n-1)$, what factor must accompany it in the trace term?
Why is it generally incorrect to compare covariance candidates using only their determinants?

Matrix Log-Likelihood for Unknown Mean and Covariance

What You'll Learn