The Finite-Sample Cost of Estimating Covariance

Why Covariance Estimation Has a Finite-Sample Cost

A mean test with known covariance can standardize the sample mean using the true covariance matrix. In practice, that matrix often has to be estimated from the same observations. This changes more than a plug-in number: it changes the exact null calibration and the distribution of the test statistic under alternatives.

The comparison below concerns the one-sample Gaussian test of $H_0:\mu=\mu_0$, where $\mu$ is a $p$-dimensional mean vector. Both tests use the same significance level and the same underlying standardized effect. Their difference is the finite-sample uncertainty introduced by estimating covariance. Assume throughout that the observations are independent and identically distributed as $N_p(\mu,\Sigma)$, with $\Sigma$ positive definite, and that $n>p$.

Definition: Write $\bar X$ for the sample mean and $S=(n-1)^{-1}\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'$ for the unbiased sample covariance matrix. The squared covariance-standardized effect and its sample noncentrality are $$ \Delta^2=(\mu-\mu_0)'\Sigma^{-1}(\mu-\mu_0), \qquad \lambda=n\Delta^2. $$ The known-covariance statistic is $Q=n(\bar X-\mu_0)'\Sigma^{-1}(\bar X-\mu_0)$. The estimated-covariance statistic is $T^2=n(\bar X-\mu_0)'S^{-1}(\bar X-\mu_0)$.

The effect $\Delta^2$ measures mean displacement in units determined by the true covariance: shifts along high-variance directions count less than equally sized shifts along low-variance directions. The value $\lambda$ is the noncentrality parameter for the sample mean. It is the same for both procedures, so differences in their exact power arise from their reference distributions and the covariance estimate, not from defining the effect differently.

Exact Calibration and Power

Let $\alpha$ be the test size, and let $\chi^2_{p,1-\alpha}$ and $F_{p,n-p;1-\alpha}$ denote the $1-\alpha$ quantiles of the central chi-square and central F distributions, respectively. Write $\chi^2_p(\lambda)$ and $F_{p,n-p}(\lambda)$ for the corresponding noncentral distributions. A noncentral F variable is the ratio $[U/p]/[V/(n-p)]$, where $U\sim\chi^2_p(\lambda)$, $V\sim\chi^2_{n-p}$, and $U,V$ are independent.

Theorem: Under the stated Gaussian model, the level-$\alpha$ known-covariance test rejects when $Q>\chi^2_{p,1-\alpha}$, and its exact power is $$ \pi_{\mathrm{known}}(\lambda) =P\{\chi^2_p(\lambda)>\chi^2_{p,1-\alpha}\}. $$ The level-$\alpha$ estimated-covariance Hotelling test rejects when $$ T^2>\frac{p(n-1)}{n-p}F_{p,n-p;1-\alpha}, $$ and its exact power is $$ \pi_{\mathrm{estimated}}(\lambda) =P\{F_{p,n-p}(\lambda)>F_{p,n-p;1-\alpha}\}. $$

For known covariance, $\sqrt n(\bar X-\mu_0)$ is normal with covariance $\Sigma$ and mean $\sqrt n(\mu-\mu_0)$. Whitening by $\Sigma^{-1/2}$ gives a $p$-dimensional normal vector with identity covariance and squared mean length $\lambda$. Its squared length therefore has distribution $\chi^2_p(\lambda)$, proving the first calibration and power formula.

For the estimated-covariance result, set $W=(n-1)S$ and $Z=\sqrt n(\bar X-\mu_0)$. Gaussian sampling implies that $Z$ and $W$ are independent, $W$ has a Wishart distribution with scale $\Sigma$ and $n-1$ degrees of freedom, and $$ T^2=(n-1)Z'W^{-1}Z. $$ Whitening both quantities reduces the argument to $W$ with identity scale and $Z\sim N_p(\eta,I_p)$, where $\eta'\eta=\lambda$.

For any fixed nonzero vector $z$, rotate coordinates so that $z/\lVert z\rVert$ is the first coordinate direction. Orthogonal invariance of the identity-scale Wishart distribution gives $$ z'W^{-1}z=\frac{\lVert z\rVert^2}{V}, \qquad V\sim\chi^2_{(n-1)-p+1}=\chi^2_{n-p}. $$ To see the degrees of freedom, represent $W=G'G$, with $G$ an $(n-1)$-by-$p$ matrix of independent standard normal entries. The reciprocal of the first diagonal entry of $W^{-1}$ is the residual sum of squares from projecting the first column of $G$ off the other $p-1$ columns. Conditional on those columns, that projection leaves $n-p$ independent standard-normal residual coordinates, so its squared length has distribution $\chi^2_{n-p}$. The conditional distribution of $V$ is the same for every direction $z$. Since $W$ is independent of $Z$, this denominator has that distribution independently of the squared length $Z'Z$, which is $\chi^2_p(\lambda)$. Consequently, $$ \frac{n-p}{p(n-1)}T^2 =\frac{Z'Z/p}{V/(n-p)} \sim F_{p,n-p}(\lambda). $$ At $\lambda=0$, this is the central F calibration; under an alternative, it yields the stated exact power. $\square$

Worked Comparisons

Worked Example: Critical Values at a Small Sample Size

Take $p=2$, $n=10$, and $\alpha=0.05$. The known-covariance test uses the 95th percentile of $\chi^2_2$, which is approximately $5.991$. Its rejection threshold for $Q$ is therefore $5.991$.

For the estimated-covariance test, the F degrees of freedom are $2$ and $8$. For $F\sim F_{2,\nu}$, its upper-tail probability is $P(F>x)=(\nu/(\nu+2x))^{\nu/2}$. Setting this equal to $0.05$ gives $$ F_{2,8;0.95}=\frac{8}{2}\left(0.05^{-2/8}-1\right) =4(20^{1/4}-1)\approx 4.459. $$

Thus the critical value on the $T^2$ scale is $$ \frac{2(10-1)}{10-2}(4.459) =\frac{18}{8}(4.459)\approx 10.033. $$ These thresholds are on different statistics and should not be interpreted as directly interchangeable cutoffs. The larger numerical threshold for $T^2$ reflects its distinct finite-sample null distribution and scaling; exact power requires comparing each statistic with its own distribution.

Worked Example: The Same Effect Under Both Procedures

Suppose $p=2$, $n=10$, $\Sigma=I_2$, and the true mean differs from $\mu_0$ by $(0.6,0.8)'$. The squared standardized effect is $$ \Delta^2=(0.6)^2+(0.8)^2=0.36+0.64=1, \qquad \lambda=n\Delta^2=10. $$

For known covariance, the exact power is $P\{\chi^2_2(10)>5.991\}$. For estimated covariance, it is $P\{F_{2,8}(10)>4.459\}$. These are exact expressions, not central-distribution approximations: both alternatives use the same $\lambda=10$, while the estimated-covariance test has only $n-p=8$ denominator degrees of freedom. The power comparison can be evaluated using the corresponding noncentral distribution functions.

The example also shows why the effect must be standardized with $\Sigma$, not with a selected sample covariance. The direction and size of the mean shift determine $\lambda$; replacing $\Sigma$ by $S$ in that definition would make the stated noncentrality random and would not describe the exact power formulas above.

Worked Example: Increasing the Sample Size

Keep $p=2$ and $\alpha=0.05$, but increase the sample size to $n=30$. The known-covariance critical value for $Q$ remains $5.991$, because its null degrees of freedom remain $2$.

For the estimated-covariance test, the F degrees of freedom are now $2$ and $28$. Using the same $F_{2,\nu}$ upper-tail formula, $$ F_{2,28;0.95} =\frac{28}{2}\left(0.05^{-2/28}-1\right) =14(20^{1/14}-1)\approx 3.340. $$ The corresponding $T^2$ cutoff is $$ \frac{2(30-1)}{30-2}(3.340) =\frac{58}{28}(3.340)\approx 6.919. $$

This is closer to the known-covariance cutoff $5.991$ than the $n=10$ cutoff of approximately $10.033$. The calculation illustrates a finite-sample effect, not a universal ranking of numerical thresholds: $Q$ and $T^2$ have different random scales. The appropriate comparison is between their powers at the same $\lambda$.

How Dimension and Sample Size Shape the Cost

The degrees of freedom $n-p$ in the estimated-covariance calibration make the dimension dependence explicit. Holding $n$ fixed while increasing $p$ reduces the denominator degrees of freedom and leaves less information for estimating a $p$-dimensional covariance matrix. As $n$ approaches $p$ from above, the sample covariance becomes increasingly unstable; at $n\leq p$, it is singular with probability one under this model, so the inverse-based statistic is not available in its stated form.

For fixed $p$, the estimated-covariance calibration approaches the known-covariance calibration as $n$ grows. Indeed, if $F_{p,n-p}$ is central, then it has the representation $$ F_{p,n-p}=\frac{U/p}{V/(n-p)},\qquad U\sim\chi^2_p,\quad V\sim\chi^2_{n-p}, $$ with $U$ and $V$ independent. As $n\to\infty$, $V/(n-p)\xrightarrow{p}1$, so $pF_{p,n-p}\xrightarrow{d}\chi^2_p$. The same argument with $U\sim\chi^2_p(\lambda)$ gives $pF_{p,n-p}(\lambda)\xrightarrow{d}\chi^2_p(\lambda)$ for fixed $\lambda$. Thus the scaled critical value and power approach their known-covariance counterparts. This limit keeps $p$ and $\lambda$ fixed; it is not a claim that the same approximation works when dimension grows with sample size.

Key takeaway: Under Gaussian sampling, exact power is determined by a noncentral chi-square distribution when covariance is known and a noncentral F distribution when covariance is estimated. The common noncentrality parameter isolates the mean effect; the F denominator degrees of freedom $n-p$ express the additional finite-sample uncertainty.

Common Mistake

A frequent error is to use the known-covariance chi-square cutoff after replacing $\Sigma$ by $S$, on the grounds that both statistics are squared Mahalanobis distances. The correction is that the estimated-covariance statistic has a random matrix in its denominator. Under the stated assumptions its exact null calibration is Hotelling’s F transformation, not $\chi^2_p$. Using the chi-square cutoff generally fails to give the nominal size at finite $n$. Another error is to say that estimating covariance changes the standardized effect itself: for the exact comparison, $\lambda=n(\mu-\mu_0)'\Sigma^{-1}(\mu-\mu_0)$ is shared by both procedures.

Check Your Understanding

Use the Gaussian model and exact calibrations developed above.

For a fixed mean shift and covariance matrix, how does $\lambda$ change when the sample size is multiplied by two?
Why does the estimated-covariance test use $n-p$ denominator degrees of freedom rather than $n-1$?
At $p=2$ and $n=10$, which reference distribution gives the known-covariance test its 5% critical value?
What happens to the estimated-covariance test’s exact calibration as $n$ grows while $p$ remains fixed?
Why is it incorrect to use the known-covariance chi-square cutoff for $T^2$ at finite sample size?