Why Estimating the Mean Changes Covariance Estimation
A covariance matrix describes variation around the population mean. If that mean is known, each observation can be centered directly. If it is unknown, the data must also estimate the center, and the same observations are then used both to locate the center and to measure variation around it. This has a finite-sample consequence: centering by the sample mean removes one direction of variation across the observations.
The effect is not just a matter of choosing a divisor. Estimating the mean changes the estimator’s expectation, its error, and the largest rank it can attain. These differences are especially relevant when the dimension \(p\) is near the sample size \(n\), because losing one rank can make a covariance estimate singular.
Two Covariance Estimators
Let \(X_1,\ldots,X_n\) be independent and identically distributed random vectors in \(\mathbb{R}^p\), with finite second moments, mean \(\mu=E[X_i]\), and covariance matrix \(\Sigma=E[(X_i-\mu)(X_i-\mu)^T]\). Write \(\bar X=n^{-1}\sum_{i=1}^n X_i\). When \(\mu\) is known, a natural estimator is the average outer product of the deviations from \(\mu\). When it is unknown, we instead center using \(\bar X\).
Here the Frobenius loss of an estimator \(A\) is \(\|A-\Sigma\|_F^2=\sum_{j=1}^p\sum_{k=1}^p(A_{jk}-\Sigma_{jk})^2\). The expectation of this loss is its Frobenius mean squared error. The factor \(n-1\) in \(S\) corrects its expectation; it does not imply that every error measure is smaller than that of \(S_0\).
Worked Example: Checking the Degrees-of-Freedom Correction
Suppose \(n=4\) observations are \(X_1=(1,0)^T\), \(X_2=(0,1)^T\), \(X_3=(-1,0)^T\), and \(X_4=(0,-1)^T\). Their mean is \(\bar X=(0,0)^T\). The sum of the four outer products is \(2I_2\), so $$ C_n=\frac{1}{4}(2I_2)=\frac{1}{2}I_2, \qquad S=\frac{1}{3}(2I_2)=\frac{2}{3}I_2. $$
The divisor \(n-1=3\) makes \(S\) the unbiased estimator under independent sampling from a population with finite covariance. It does not make this particular observed matrix equal to the unknown population covariance. Unbiasedness is a statement about the estimator’s expectation over repeated samples, not a guarantee about any one sample.
Why the Centered Estimator Uses \(n-1\)
The correction follows directly from the expectation of the centered sum of outer products. Let \(Y_i=X_i-\mu\), so \(E[Y_i]=0\), \(E[Y_iY_i^T]=\Sigma\), and \(\bar Y=n^{-1}\sum_iY_i=\bar X-\mu\). Expanding around the true mean gives $$ \sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^T =\sum_{i=1}^nY_iY_i^T-n\bar Y\bar Y^T. $$
The first term has expectation \(n\Sigma\). Independence and zero means give \(E[\bar Y\bar Y^T]=\Sigma/n\): in the expansion of \(n^{-2}\sum_{i,j}Y_iY_j^T\), each term with \(i\ne j\) has expectation zero, and the \(n\) diagonal terms each have expectation \(\Sigma\). Thus the expected centered sum is \(n\Sigma-\Sigma=(n-1)\Sigma\).
The first identity follows by dividing the expected centered sum by \(n-1\); the second follows by dividing the same sum by \(n\). This argument needs finite second moments, but not a Gaussian distribution.
Worked Example: Bias When the Mean Is Estimated
Suppose \(n=5\) and the population covariance is \(\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}\). The expectation of the divisor-\(n\) centered estimator is $$ E[C_n]=\frac{4}{5}\Sigma =\begin{pmatrix}3.2&0\\0&0.8\end{pmatrix}. $$ The bias matrix is therefore \(E[C_n]-\Sigma=-\frac{1}{5}\Sigma=\begin{pmatrix}-0.8&0\\0&-0.2\end{pmatrix}\). In contrast, \(E[S]=\Sigma\). The plug-in estimator with divisor \(n\) shrinks the expectation toward zero; using \(n-1\) removes this particular bias.
Exact Frobenius Error Under Gaussian Sampling
Unbiasedness alone does not determine estimation error. To compare errors exactly, impose the additional condition \(X_i\sim N(\mu,\Sigma)\) independently, with \(\Sigma\) positive definite, and assume \(n\ge2\). Define \(A_\Sigma=\operatorname{tr}(\Sigma^2)+\{\operatorname{tr}(\Sigma)\}^2\). For a centered Gaussian vector \(Z\sim N(0,\Sigma)\), the Gaussian fourth-moment identity gives \(E[\|Z\|^4]=\{\operatorname{tr}(\Sigma)\}^2+2\operatorname{tr}(\Sigma^2)\). Hence $$ E[\|ZZ^T-\Sigma\|_F^2] =E[\|Z\|^4]-\operatorname{tr}(\Sigma^2) =A_\Sigma. $$
For independent copies, the centered random matrices \(Z_iZ_i^T-\Sigma\) have mean zero, so the expected squared Frobenius norm of their average is \(1/n\) times the expected squared norm of one such matrix: cross terms have expectation zero. This proves the known-mean risk formula. For the estimated-mean case, an orthogonal transformation of the \(n\) observation rows separates the sample-mean direction from \(n-1\) orthogonal residual directions. Under Gaussian sampling, the transformed rows are independent \(N(0,\Sigma)\) vectors. The centered sum of outer products is the sum of the outer products from those \(n-1\) residual rows. Applying the same calculation completes the comparison.
For the last identity, write \(C_n=\frac{n-1}{n}S\), so \(C_n-\Sigma=\frac{n-1}{n}(S-\Sigma)-\frac{1}{n}\Sigma\). Since \(E[S-\Sigma]=0\), the cross term has expectation zero. Therefore the risk is \(\frac{(n-1)^2}{n^2}\frac{A_\Sigma}{n-1}+\frac{\operatorname{tr}(\Sigma^2)}{n^2}\), which simplifies to the displayed expression. The term \(\|\Sigma\|_F^2=\operatorname{tr}(\Sigma^2)\) is the squared bias contribution.
Worked Example: Comparing Gaussian Risks Numerically
Let \(n=5\) and \(\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}\). Then \(\operatorname{tr}(\Sigma)=5\), \(\operatorname{tr}(\Sigma^2)=17\), and \(A_\Sigma=17+25=42\). The known-mean estimator has risk \(42/5=8.4\); the unbiased centered estimator has risk \(42/4=10.5\). For \(C_n\), the risk is $$ \frac{5(17)+4(25)}{5^2}=\frac{185}{25}=7.4. $$
Thus \(C_n\) has lower Frobenius risk than \(S_0\) in this example, despite being biased. This is not a contradiction: correcting bias does not necessarily minimize mean squared error. Here the bias of \(C_n\) reduces its variance enough to yield lower total risk. The comparison is specific to this loss and model; it does not establish that the divisor-\(n\) estimator is always preferable.
Rank Loss When Dimension Approaches Sample Size
A separate effect of estimating the mean is visible without Gaussian assumptions. Arrange the centered observations as the rows of an \(n\times p\) matrix. Centering amounts to multiplying by the matrix \(H=I_n-\frac{1}{n}\mathbf{1}\mathbf{1}^T\), where \(\mathbf{1}\) is the \(n\)-vector of ones. This matrix has rank \(n-1\), because it removes the direction \(\mathbf{1}\). Consequently, $$ \operatorname{rank}(S)\leq \min(p,n-1). $$ For the known-mean estimator, the corresponding matrix of deviations from \(\mu\) has \(n\) rows, so \(\operatorname{rank}(S_0)\leq\min(p,n)\). If \(\Sigma\) is positive definite and the observations are Gaussian, these upper bounds are attained with probability one. In particular, for \(p\ge n\), \(S\) is necessarily singular; when \(p=n\), \(S_0\) can be full rank while \(S\) cannot.
Worked Example: One Lost Rank at the Boundary
Take \(p=n=6\) and assume a nonsingular Gaussian population covariance. The known-mean estimator is based on six independent deviation vectors, so \(\operatorname{rank}(S_0)=6\) with probability one. Centering the six observations imposes the exact relation that their six deviations from \(\bar X\) sum to zero. Thus \(\operatorname{rank}(S)\leq5\), and in this Gaussian setting the rank is 5 with probability one.
Multiplying \(S\) by \(n/(n-1)=6/5\) changes its eigenvalues but not its rank. The degrees-of-freedom correction therefore cannot make the centered estimate invertible in this example. When \(p\) is near \(n\), that distinction can matter for procedures that require an inverse covariance matrix.
What to Conclude—and a Common Pitfall
There are three different comparisons to keep separate. First, \(S\) is unbiased under independent sampling with finite covariance, whereas \(C_n\) is biased downward by the factor \((n-1)/n\). Second, under Gaussian sampling, the unbiased centered estimator has larger exact Frobenius risk than the known-mean estimator by the factor \(n/(n-1)\). Third, centered data have at most \(n-1\) independent row directions, so covariance rank can be limited even after rescaling. The risk formulas also show why the dimension matters: for \(\Sigma=I_p\), \(A_\Sigma=p+p^2\), and the known-mean and unbiased centered risks are \((p+p^2)/n\) and \((p+p^2)/(n-1)\), respectively.
Common mistake: “The sample covariance is unbiased because I divided by \(n-1\), so the correction also fixes the loss of information from estimating the mean.” The first part is correct under the stated sampling conditions; the second does not follow. Unbiasedness is an expectation property, while rank is determined by the geometry of the centered data matrix. Multiplying by a nonzero scalar changes neither that geometry nor the rank. Another mistake is to infer from the Gaussian risk comparison that the biased estimator \(C_n\) is always best. That conclusion requires a specified loss and distribution; the calculation above establishes it only for the cases actually compared.
Check Your Understanding
Use the definitions and conditions in this tutorial to answer the following.
- Why is \(S\) unbiased with divisor \(n-1\) when the observations have finite covariance?
- Under Gaussian sampling, how do the exact Frobenius risks of \(S_0\) and \(S\) compare?
- For \(p=n\), what rank bound applies to the centered sample covariance, and why?
- Does multiplying a singular centered covariance estimate by \(n/(n-1)\) make it nonsingular? Explain.
- Can an estimator with nonzero bias have smaller Frobenius mean squared error than an unbiased estimator? Give the relevant reason.