Profiling Out An Unknown Mean

Why an Unknown Mean Changes the Covariance Test

A covariance test asks whether the observed spread and dependence among coordinates are compatible with a specified covariance matrix. If the population mean is also unknown, it must be estimated under both the null model and the unrestricted model. This has a direct consequence: covariance is assessed around the fitted mean, so the relevant scatter matrix is centered rather than formed around the origin.

That centering removes one degree of freedom. It does not, however, mean that every occurrence of the sample size \(n\) in the likelihood-ratio formula should be replaced by \(n-1\). The likelihood uses all \(n\) observations; the centered scatter has a Wishart distribution with \(n-1\) degrees of freedom. Keeping these two roles separate is essential.

The notation \(A\) will denote the centered scatter matrix. The conventional unbiased sample covariance is \(S=A/(n-1)\); it is not the maximum-likelihood estimate in this model. When \(n>p\), \(A\) is positive definite with probability one under a nonsingular multivariate normal model, and the unrestricted covariance likelihood has a finite maximum. The derivation below assumes this condition.

Profiling the Mean and Maximizing the Covariance Likelihood

The joint likelihood, apart from its usual normalizing constant, is \(L(\mu,\Sigma)\propto|\Sigma|^{-n/2}\exp\{-\frac12\sum_i(X_i-\mu)'\Sigma^{-1}(X_i-\mu)\}\). For a fixed positive-definite \(\Sigma\), the quadratic term can be decomposed exactly:

The second term on the right is nonnegative and equals zero only at \(\mu=\bar X\). Thus, for every fixed covariance matrix, the likelihood is maximized over the mean at the sample mean. This holds under the null and under the unrestricted alternative, so profiling out the mean leaves the same centered scatter matrix in both likelihoods.

To verify that this is a global maximum, take any positive-definite \(\Sigma\) and let \(b_1,\ldots,b_p\) be the eigenvalues of \(B=\Sigma^{-1/2}\widehat\Sigma_{\mathrm{MLE}}\Sigma^{-1/2}\). Since \(A=n\widehat\Sigma_{\mathrm{MLE}}\), subtracting the profiled log-likelihood at \(\widehat\Sigma_{\mathrm{MLE}}\) from the one at \(\Sigma\) gives

The inequality follows from \(\log t\leq t-1\) for \(t>0\), with equality only at \(t=1\). Hence the displayed estimate is the unique unrestricted maximizer. Proof complete. \(\square\)

Under \(H_0\), the profiled likelihood is evaluated at \(\Sigma_0\); under the alternative, it is evaluated at \(A/n\). Define the likelihood ratio as the maximized null likelihood divided by the maximized unrestricted likelihood, so \(0\leq\Lambda\leq1\).

Indeed, the profiled likelihood at the null is proportional to \( |\Sigma_0|^{-n/2}\exp\{-\operatorname{tr}(\Sigma_0^{-1}A)/2\} \). At the unrestricted maximum it is proportional to \( |A/n|^{-n/2}\exp\{-np/2\} \). Taking their ratio proves the expression for \(\Lambda\). Taking \(-2\) times its logarithm gives \( n\log(|\Sigma_0|/|\widehat\Sigma_{\mathrm{MLE}}|)+\operatorname{tr}(\Sigma_0^{-1}A)-np \). Since \(A=n\widehat\Sigma_{\mathrm{MLE}}\) and \( |\Sigma_0^{-1}\widehat\Sigma_{\mathrm{MLE}}|=|\widehat\Sigma_{\mathrm{MLE}}|/|\Sigma_0| \), this is precisely the stated log form. Proof complete. \(\square\)

The formula can also be expressed using the unbiased sample covariance \(S=A/(n-1)\). Substitution gives

The factor \((n-1)/n\) appears because the likelihood maximizer is \(A/n\), not \(S\). Dropping that factor while using \(S\) changes the statistic.

Why the Centered Scatter Has \(n-1\) Degrees of Freedom

For clarity, a Wishart distribution \(W_p(\Sigma,\nu)\) is the distribution of a sum of \(\nu\) independent outer products \(Y_jY_j'\), where each \(Y_j\sim N_p(0,\Sigma)\). In the present model, the centered scatter has distribution \(A\sim W_p(\Sigma,n-1)\). The reason is that centering removes the component in the direction of the sample mean.

To see this, arrange the centered observations around \(\mu\) as the rows of an \(n\times p\) matrix \(Z\). Choose an orthogonal \(n\times n\) matrix \(Q\) whose first row is \(n^{-1/2}(1,\ldots,1)\), and write \(Y=QZ\). Orthogonal transformation preserves the joint normal distribution across the rows, so the rows of \(Y\) are independent \(N_p(0,\Sigma)\) vectors. The first row represents the sample-mean direction. The remaining rows represent the \(n-1\) orthogonal residual directions. Since \(I_n-\mathbf{1}\mathbf{1}'/n\) projects onto those residual directions,

This is a sum of \(n-1\) independent normal outer products, which proves the claimed Wishart distribution. \(\square\)

The likelihood-ratio expression and the scatter distribution therefore involve \(n\) and \(n-1\) for different reasons. The exponent \(n/2\) in the likelihood comes from the \(n\) independent observations. The \(n-1\) degrees of freedom arise from the \(n-1\) residual directions left after estimating the mean. The likelihood maximizer is \(A/n\), whereas \(A/(n-1)\) is unbiased for \(\Sigma\).

Worked Calculations

Common Pitfalls and Practical Interpretation

The statistic measures how far the fitted covariance is from the specified covariance on a scale that accounts for both determinant and trace. The determinant term reflects overall volume, while the trace term reflects the total standardized variation relative to \(\Sigma_0\). Their combination is nonnegative: if \(b_1,\ldots,b_p\) are the eigenvalues of \(\Sigma_0^{-1/2}\widehat\Sigma_{\mathrm{MLE}}\Sigma_0^{-1/2}\), then \( -2\log\Lambda=n\sum_j(b_j-\log b_j-1)\geq0 \), because \(t-\log t-1\geq0\) for \(t>0\). Equality holds exactly when \(\widehat\Sigma_{\mathrm{MLE}}=\Sigma_0\).

• Using uncentered observations. Treating \(\sum_iX_iX_i'\) as scatter when the mean is unknown incorrectly forces the reference point to zero. The fitted mean is \(\bar X\), so use \(A=\sum_i(X_i-\bar X)(X_i-\bar X)'\).
• Replacing \(n\) everywhere by \(n-1\). Estimating the mean changes the scatter’s degrees of freedom, not the number of independent observations in the likelihood. Keep the likelihood exponent \(n/2\), and use \(A/n\) for the unrestricted MLE.
• Using the unbiased covariance as the MLE. \(S=A/(n-1)\) is unbiased, but the likelihood is maximized at \(A/n\). If writing the statistic with \(S\), retain the factor \((n-1)/n\).
• Ignoring a singular scatter matrix. If \(n\leq p\), the centered scatter is singular. The unrestricted positive-definite covariance likelihood then has no finite maximum, so the ordinary likelihood-ratio derivation above does not apply.

For a specified covariance test, the central consequence of an unknown mean is therefore precise: profiling the mean replaces origin-based scatter with centered scatter, and the latter has \(n-1\) residual degrees of freedom. The likelihood-ratio formula follows from that centered scatter, while its likelihood exponent continues to reflect the full sample size \(n\).