Power with Estimated Covariance

Why Estimated Covariance Changes the Power Calculation

When the covariance matrix is known, the uncertainty in a sample mean can be standardized directly. When covariance is estimated from the same sample, the test statistic also varies with the sample covariance. Its exact power therefore must account for both sources of variation. Under multivariate normal sampling, this is done using the noncentral F distribution, rather than the central F distribution used to calibrate the null test.

The calculation has three parts: specify the mean shift and covariance under the alternative, convert them to a noncentrality parameter, and evaluate the noncentral F upper tail at the test's central-F critical value. The noncentral F tail can itself be evaluated as a weighted sum of central F tails. Keeping the scaling in that sum correct is essential.

Test Statistic and Exact Distribution

Let $X_1,\ldots,X_n$ be independent $p$-dimensional observations from $N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite and unknown. Let $\bar X$ be the sample mean and let $S$ be the sample covariance matrix with divisor $n-1$. For testing $H_0:\mu=\mu_0$, define Hotelling's statistic

T^2=n(\bar X-\mu_0)^\mathsf{T}S^{-1}(\bar X-\mu_0).

Assume $n>p$, so $S$ is invertible with probability one under this model. The usual level-$\alpha$ test rejects for large values of $T^2$, or equivalently for large values of the rescaled statistic

F_{\mathrm{obs}}=\frac{n-p}{p(n-1)}T^2.

Write $F_{a,q}(x)$ for the cumulative distribution function of a central $F$ random variable with numerator degrees of freedom $a$ and denominator degrees of freedom $q$, and write $F^{-1}_{a,q}(u)$ for its $u$-quantile.

Theorem: Under the stated independent multivariate normal model, the statistic $F_{\mathrm{obs}}$ has a central $F_{p,n-p}$ distribution when $\mu=\mu_0$. Under a fixed alternative $\mu\ne\mu_0$, it has a noncentral $F$ distribution with numerator degrees of freedom $p$, denominator degrees of freedom $n-p$, and noncentrality parameter $$ \lambda=n(\mu-\mu_0)^\mathsf{T}\Sigma^{-1}(\mu-\mu_0). $$ Thus the level-$\alpha$ test rejects when $F_{\mathrm{obs}}>c_\alpha$, where $c_\alpha=F^{-1}_{p,n-p}(1-\alpha)$.

This exact noncentral Hotelling distribution requires multivariate normal sampling and a nonsingular sample covariance. The noncentrality parameter uses the true covariance $\Sigma$, not the observed sample covariance $S$. The sample covariance's randomness is already incorporated in the noncentral F distribution.

Exact Power and the Scaled Mixture Formula

For a specified alternative, power is the probability of rejection when that alternative is true. The theorem immediately gives the exact expression

\pi(\mu,\Sigma) =P_{\mu,\Sigma}(F_{\mathrm{obs}}>c_\alpha) =1-F_{p,n-p,\lambda}(c_\alpha), \qquad \lambda=n(\mu-\mu_0)^\mathsf{T}\Sigma^{-1}(\mu-\mu_0),

where $F_{p,q,\lambda}$ denotes the noncentral F cumulative distribution function. For calculations, its upper tail has a Poisson-mixture representation. Define $\bar F_{a,q}(x)=1-F_{a,q}(x)$. If $J$ has a Poisson distribution with mean $\lambda/2$, the defining chi-square mixture for a noncentral chi-square variable gives

P(J=j)=e^{-\lambda/2}\frac{(\lambda/2)^j}{j!}, \qquad Z\mid J=j\sim\chi^2_{p+2j}.

A noncentral F variable is $(Z/p)/(V/q)$, where $V\sim\chi^2_q$ is independent of $Z$. Conditional on $J=j$, $Z/(p+2j)$ and $V/q$ form a central $F_{p+2j,q}$ ratio. But the noncentral F numerator is $Z/p$, not $Z/(p+2j)$. Therefore the conditional noncentral F variable is $((p+2j)/p)F_{p+2j,q}$. This distinction determines the argument of each central F tail.

Result: For $c>0$, the exact noncentral F upper tail is $$ 1-F_{p,q,\lambda}(c) =\sum_{j=0}^{\infty}e^{-\lambda/2}\frac{(\lambda/2)^j}{j!} \bar F_{p+2j,q}\!\left(\frac{cp}{p+2j}\right). $$ The factor $p/(p+2j)$ in the central F tail argument is required.

To verify the result, condition on $J=j$. The conditional event that the noncentral F variable exceeds $c$ is

\frac{p+2j}{p}F_{p+2j,q}>c \quad\Longleftrightarrow\quad F_{p+2j,q}>\frac{cp}{p+2j}.

Its conditional probability is the central F tail at $cp/(p+2j)$. Averaging these conditional probabilities over the Poisson probabilities gives the displayed series. The weights are nonnegative and sum to one, so the series converges and represents the desired probability exactly.

Worked Power Calculations

Worked Example: A Closed-Form Check in Two Dimensions

Suppose $p=2$, $n=4$, $\alpha=0.05$, $\Sigma=I_2$, and the specified alternative is $\mu-\mu_0=(1/\sqrt{2},0)^\mathsf{T}$. Calculate the exact power.

Here $q=n-p=2$, and the noncentrality is

\lambda=4(1/\sqrt{2},0)I_2(1/\sqrt{2},0)^\mathsf{T} =4\left(\frac12\right)=2.

For $F_{2,2}$, the upper tail is $1/(1+c)$, so $c_{0.05}=19$. In the mixture, let $J\sim\operatorname{Poisson}(1)$. The $j$th central-F component has degrees of freedom $(2+2j,2)$, and its correctly scaled argument is $19\cdot2/(2+2j)=19/(j+1)$. For $F_{2k,2}$, its upper tail at $x$ is $1-(1+1/(kx))^{-k}$. Taking $k=j+1$ and $x=19/(j+1)$ gives a conditional tail of $1-(19/20)^{j+1}$. Hence

\begin{aligned} \pi &=E\left[1-\left(\frac{19}{20}\right)^{J+1}\right]\\ &=1-\frac{19}{20}E\left[\left(\frac{19}{20}\right)^J\right]\\ &=1-\frac{19}{20}\exp\left(\frac{19}{20}-1\right)\\ &=1-\frac{19}{20}e^{-1/20} \approx 0.09633. \end{aligned}

The power is about $9.63\%$, above the $5\%$ significance level. This example also checks the scaling: the $j=1$ component is $2F_{4,2}$, so its central F tail is evaluated at $19/2$, not at 19.

Worked Example: A Two-Variable Alternative with Ten Residual Degrees of Freedom

Let $p=2$, $n=12$, $\alpha=0.05$, $\Sigma=I_2$, and $\mu-\mu_0=(1/\sqrt{2},0)^\mathsf{T}$. The residual degrees of freedom are $q=10$, and

\lambda=12\left(\frac12\right)=6, \qquad \mu_J=\lambda/2=3, \qquad c=F^{-1}_{2,10}(0.95).

The exact power is a Poisson-weighted sum. The first three terms, with their degrees of freedom and scaled arguments shown, are

\begin{aligned} \pi =e^{-3}\bigg[& \bar F_{2,10}(c) +3\bar F_{4,10}\left(\frac{c}{2}\right) +\frac{9}{2}\bar F_{6,10}\left(\frac{c}{3}\right)\\ &+\sum_{j=3}^{\infty}\frac{3^j}{j!} \bar F_{2+2j,10}\left(\frac{c}{j+1}\right) \bigg]. \end{aligned}

For example, the $j=1$ term has weight $e^{-3}3$; its conditional statistic is $2F_{4,10}$, which explains the argument $c/2$. The $j=0$ term equals $e^{-3}(0.05)$, since $\bar F_{2,10}(c)=0.05$ by the definition of $c$. The displayed series is the exact power; evaluating its central F tails gives a numerical value to any desired precision.

Worked Example: A Three-Variable Alternative

Suppose $p=3$, $n=15$, $\alpha=0.10$, $\Sigma=I_3$, and $\mu-\mu_0=(1,1/2,0)^\mathsf{T}$. Then $q=12$, and the squared Euclidean length of the mean shift is $1+1/4=5/4$. Thus

\lambda=15\left(\frac54\right)=\frac{75}{4}=18.75, \qquad \lambda/2=9.375, \qquad c=F^{-1}_{3,12}(0.90).

The exact power is

\begin{aligned} \pi =e^{-9.375}\bigg[& \bar F_{3,12}(c) +9.375\,\bar F_{5,12}\left(\frac{3c}{5}\right)\\ &+\frac{(9.375)^2}{2}\, \bar F_{7,12}\left(\frac{3c}{7}\right)\\ &+\sum_{j=3}^{\infty}\frac{(9.375)^j}{j!} \bar F_{3+2j,12}\left(\frac{3c}{3+2j}\right) \bigg]. \end{aligned}

The first component is the null-calibration tail, $\bar F_{3,12}(c)=0.10$. For $j=1$, the conditional statistic is $(5/3)F_{5,12}$, so the central F argument is $3c/5$. For $j=2$, it is $(7/3)F_{7,12}$, giving argument $3c/7$. These components make explicit how the random numerator degrees of freedom in the mixture affect the tail calculation.

Truncation, Interpretation, and a Common Pitfall

The Poisson series is infinite, but it can be evaluated accurately by truncation. If terms through $j=K$ are retained, the omitted contribution is nonnegative and no larger than the omitted Poisson probability:

0\le \sum_{j=K+1}^{\infty}P(J=j)\, \bar F_{p+2j,q}\left(\frac{c_\alpha p}{p+2j}\right) \le P(J>K), \qquad J\sim\operatorname{Poisson}(\lambda/2).

The bound follows because every central F tail is between zero and one. A computation can therefore choose $K$ so that the Poisson upper tail is below a specified tolerance. This provides a direct error check for a numerical power calculation.

Common mistake: treating every mixture component as an unscaled central F variable. That reasoning changes the numerator divisor from $p$ to $p+2j$. For $j=1$, it incorrectly uses $F_{p+2,q}$ at $c_\alpha$; the actual conditional statistic is $((p+2)/p)F_{p+2,q}$, so its tail must be evaluated at $c_\alpha p/(p+2)$. Omitting this scaling understates the component's tail probability and produces an incorrect power.

Power depends on the alternative through its covariance-standardized distance from the null, $\lambda=n(\mu-\mu_0)^\mathsf{T}\Sigma^{-1}(\mu-\mu_0)$. The exact distribution accounts for estimation of covariance, while the noncentrality describes the specified alternative in the units set by the true covariance. Thus the inputs should be a stated mean, a stated covariance model, a sample size, and a significance level—not a covariance estimate substituted into $\lambda$ after observing the data.

Check Your Understanding

Use the definitions and mixture formula to answer the following.

For $p=4$, $n=20$, and a specified alternative, write the numerator and denominator degrees of freedom and the expression for the noncentrality parameter.
Why is the $j=1$ conditional component in the noncentral F mixture not simply $F_{p+2,q}$?
For $p=2$, what central F tail argument corresponds to mixture index $j=3$ and critical value $c_\alpha$?
Which covariance matrix belongs in the noncentrality parameter, and why is the sample covariance not substituted there?
How can the omitted Poisson probability bound the error when a mixture calculation is truncated?