Gaussian Covariance Estimation With Known Mean

Why Fixing the Mean Changes the Covariance Estimate

A covariance matrix describes variation around a specified center. If that center is estimated from the same observations, the observed deviations sum to zero. If the center is known in advance, the observations need not balance around it: their shared displacement from the known center contributes to the covariance estimate. This distinction changes the scatter matrix in the likelihood and can even change whether a maximum-likelihood estimate (MLE) exists.

Suppose $X_1,\ldots,X_n$ are independent observations in $\mathbb{R}^p$, with $X_i\sim N_p(\mu_0,\Sigma)$, where $\mu_0$ is fixed and known and $\Sigma$ is an unknown positive-definite covariance matrix. The task here is to estimate $\Sigma$, not to estimate or optimize over the mean.

Definition: For observed vectors $x_1,\ldots,x_n$ and known mean $\mu_0$, define the known-mean scatter matrix $$ A_0=\sum_{i=1}^n (x_i-\mu_0)(x_i-\mu_0)^T. $$ When $A_0$ is positive definite, the Gaussian covariance MLE with known mean is $$ \hat{\Sigma}_{\mathrm{known}}=\frac{1}{n}A_0. $$ If $A_0$ is singular, the likelihood is unbounded over positive-definite $\Sigma$, so there is no finite covariance MLE.

Deriving the Likelihood Estimate

The multivariate normal density for one observation is proportional to $|\Sigma|^{-1/2}\exp\{-\tfrac12(x_i-\mu_0)^T\Sigma^{-1}(x_i-\mu_0)\}$. Independence gives the likelihood for the full sample. Using $\operatorname{tr}(BC)=\operatorname{tr}(CB)$ to write the sum of quadratic forms as a trace, the log-likelihood is

\ell(\Sigma)=-\frac{np}{2}\log(2\pi)-\frac{n}{2}\log|\Sigma|-\frac{1}{2}\operatorname{tr}(\Sigma^{-1}A_0).

The parameter space consists of positive-definite matrices. First suppose $A_0$ is positive definite. Define $B=A_0^{1/2}\Sigma^{-1}A_0^{1/2}$, which is also positive definite. The determinant and trace identities give $|\Sigma|=|A_0|/|B|$ and $\operatorname{tr}(\Sigma^{-1}A_0)=\operatorname{tr}(B)$. If $\lambda_1,\ldots,\lambda_p$ are the positive eigenvalues of $B$, the part of the log-likelihood that depends on $B$ is

\frac{n}{2}\log|B|-\frac{1}{2}\operatorname{tr}(B) =\sum_{j=1}^p\left(\frac{n}{2}\log\lambda_j-\frac{1}{2}\lambda_j\right).

For a positive scalar $\lambda$, the derivative of $g(\lambda)=\tfrac{n}{2}\log\lambda-\tfrac12\lambda$ is $g'(\lambda)=n/(2\lambda)-1/2$. It is positive for $\lambda<n$, zero at $n$, and negative for $\lambda>n$; also $g''(\lambda)=-n/(2\lambda^2)<0$. Thus each term has its unique global maximum at $\lambda=n$. The matrix objective is maximized when every eigenvalue of $B$ equals $n$, which means $B=nI_p$. Solving $A_0^{1/2}\Sigma^{-1}A_0^{1/2}=nI_p$ yields the estimate.

\Sigma^{-1}=nA_0^{-1} \qquad\Longrightarrow\qquad \hat{\Sigma}_{\mathrm{known}}=\frac{A_0}{n}.

This argument proves both global optimality and uniqueness when $A_0$ is positive definite. It does not depend on a local stationary point being the global maximum.

Worked Example: A One-Dimensional Known-Mean Estimate

Let $X_1,X_2,X_3$ be independent $N(\mu_0,\sigma^2)$ observations with known mean $\mu_0=0$, and suppose the observed values are $1,2,4$. The known-mean scatter is the sum of squared deviations from zero:

$$ A_0=(1-0)^2+(2-0)^2+(4-0)^2=1+4+16=21. $$

Since $A_0>0$, the MLE exists. Dividing by the sample size gives

\hat{\sigma}^2_{\mathrm{known}}=\frac{21}{3}=7.

The center in this calculation is the specified value $0$, not the observed sample mean $7/3$. Replacing the known center by the sample mean would answer a different estimation problem.

Existence: When the Scatter Is Singular

The known-mean scatter $A_0$ is always positive semidefinite, but it need not be positive definite. Its rank is the dimension of the span of the residual vectors $x_i-\mu_0$. If those vectors do not span $\mathbb{R}^p$, the data leave at least one direction with zero observed scatter. In that case, the likelihood can increase without bound by making the fitted variance in a zero-scatter direction arbitrarily small.

To verify this, suppose $A_0$ has rank $r<p$. Choose an orthonormal basis in which $A_0$ is diagonal, with $r$ positive eigenvalues followed by $p-r$ zeros. In that same basis, let $\Sigma_\varepsilon$ have eigenvalue $1$ in the first $r$ directions and eigenvalue $\varepsilon>0$ in the remaining directions. Then $|\Sigma_\varepsilon|=\varepsilon^{p-r}$, while $\operatorname{tr}(\Sigma_\varepsilon^{-1}A_0)$ is constant in $\varepsilon$. The likelihood therefore has the form $C\varepsilon^{-n(p-r)/2}$ for a constant $C>0$, and tends to infinity as $\varepsilon$ decreases to zero. Every $\Sigma_\varepsilon$ is positive definite, but there is no finite maximizing covariance matrix.

Worked Example: Singular Scatter and an Unbounded Likelihood

Take $p=2$, known mean $\mu_0=(0,0)^T$, and observations $x_1=(1,0)^T$, $x_2=(2,0)^T$. Both residuals lie on the horizontal axis, and

A_0= \begin{pmatrix} 1&0\\ 0&0 \end{pmatrix} + \begin{pmatrix} 4&0\\ 0&0 \end{pmatrix} = \begin{pmatrix} 5&0\\ 0&0 \end{pmatrix}.

For $\Sigma_\varepsilon=\operatorname{diag}(1,\varepsilon)$, with $\varepsilon>0$, the determinant is $\varepsilon$ and the trace term is

\operatorname{tr}(\Sigma_\varepsilon^{-1}A_0) =\operatorname{tr} \begin{pmatrix} 1&0\\ 0&1/\varepsilon \end{pmatrix} \begin{pmatrix} 5&0\\ 0&0 \end{pmatrix} =5.

Here $n=2$, so the likelihood is proportional to $\varepsilon^{-n/2}\exp(-5/2)=\varepsilon^{-1}\exp(-5/2)$. It tends to infinity as $\varepsilon\to0$. The likelihood has no finite maximum within the positive-definite covariance parameter space.

Consequently, a finite MLE exists exactly when the observed residuals about the known mean span all $p$ dimensions. Since there are $n$ residual vectors, $n\ge p$ is necessary, though it is not by itself sufficient for a particular data set: the residuals must actually span $\mathbb{R}^p$.

Contrast With an Unknown Mean

When the mean is unknown, the unrestricted Gaussian MLE uses the sample mean $\bar{x}=n^{-1}\sum_{i=1}^n x_i$ and the centered scatter

A_c=\sum_{i=1}^n(x_i-\bar{x})(x_i-\bar{x})^T, \qquad \hat{\Sigma}_{\mathrm{unknown}}=\frac{A_c}{n},

provided $A_c$ is positive definite. The key comparison is an exact decomposition. Write $d=\bar{x}-\mu_0$ and expand each residual as $x_i-\mu_0=(x_i-\bar{x})+d$. The cross terms sum to zero because $\sum_i(x_i-\bar{x})=0$. Therefore

A_0=A_c+n(\bar{x}-\mu_0)(\bar{x}-\mu_0)^T.

Thus known-mean scatter equals centered scatter plus a positive-semidefinite rank-at-most-one term recording the sample mean's displacement from the known mean. The two covariance estimates agree when $\bar{x}=\mu_0$, but otherwise the known-mean estimate includes that extra contribution. Notice also that centering residuals imposes a linear dependence among them, so $A_c$ has rank at most $n-1$; by contrast, $A_0$ can have rank $p$ when $n=p$.

Worked Example: Comparing the Two Scatter Matrices

Let $p=2$, let the known mean be $\mu_0=(0,0)^T$, and observe $x_1=(2,0)^T$, $x_2=(1,1)^T$, and $x_3=(0,0)^T$. The known-mean scatter is

A_0= \begin{pmatrix}4&0\\0&0\end{pmatrix} +\begin{pmatrix}1&1\\1&1\end{pmatrix} +\begin{pmatrix}0&0\\0&0\end{pmatrix} = \begin{pmatrix}5&1\\1&1\end{pmatrix}.

Its determinant is $5(1)-1(1)=4>0$, so it is positive definite. The known-mean MLE is

\hat{\Sigma}_{\mathrm{known}}=\frac{A_0}{3} =\begin{pmatrix}5/3&1/3\\1/3&1/3\end{pmatrix}.

The sample mean is $\bar{x}=(1,1/3)^T$. The decomposition gives

3\bar{x}\bar{x}^T = \begin{pmatrix}3&1\\1&1/3\end{pmatrix}, \qquad A_c=A_0-3\bar{x}\bar{x}^T = \begin{pmatrix}2&0\\0&2/3\end{pmatrix}.

The centered scatter is positive definite, and the unknown-mean MLE is

\hat{\Sigma}_{\mathrm{unknown}}=\frac{A_c}{3} =\begin{pmatrix}2/3&0\\0&2/9\end{pmatrix}.

The difference between the two MLEs is $\bar{x}\bar{x}^T$, as the scatter decomposition requires. In this example, estimating the center removes the contribution due to the observed mean's displacement from $(0,0)^T$.

Interpretation and Common Pitfall

The divisor in the Gaussian MLE is $n$, because the likelihood maximization gives $A_0/n$. This is not a claim that the MLE is the only useful covariance estimator. With a known mean, $E[A_0]=n\Sigma$, since each residual has mean zero and covariance $\Sigma$; consequently, $A_0/n$ is unbiased as well. When the mean is estimated, the centered scatter has a different expectation, and the MLE's denominator remains $n$. Questions about unbiased estimation and maximum likelihood are distinct.

Common mistake: use the sample mean even though the mean is known. This substitutes $A_c$ for $A_0$ and discards the mean-displacement term. Use the stipulated $\mu_0$ in every residual.
Common mistake: divide the known-mean scatter by $n-1$ to obtain the MLE. The likelihood derivation gives $A_0/n$; $n-1$ is not the MLE denominator in this model.
Common mistake: assume a covariance MLE always exists. If $A_0$ is singular, the likelihood is unbounded, not maximized by an ordinary positive-definite covariance matrix.

The practical distinction is simple but consequential: covariance is measured relative to the model's center. A fixed, known center keeps all deviations from that center in the scatter; an estimated center shifts to the sample and produces a centered scatter instead.

Check Your Understanding

Use the fixed-mean likelihood and scatter decomposition to answer the following.

For known $\mu_0$, write the scatter matrix whose value determines the covariance MLE. What divisor gives the MLE?
Why does a singular known-mean scatter make the likelihood unbounded over positive-definite covariance matrices?
State the decomposition relating $A_0$, $A_c$, and $\bar{x}-\mu_0$. Under what condition are the two scatters equal?
Why is $n\ge p$ necessary but not sufficient for a finite known-mean covariance MLE?
Explain why replacing $n$ by $n-1$ does not give the Gaussian maximum-likelihood estimate.