Power Calculations for Multi-Parameter F Tests

From a Multi-Parameter Hypothesis to Power

A multi-parameter F test asks whether several linear restrictions on a model’s parameters hold together. Its power is the probability of rejecting that joint null hypothesis when a particular alternative is true. Unlike a large-sample approximation, an exact finite-sample calculation uses the sampling distribution of the test statistic at the specified alternative.

Consider the full-rank normal linear model \(Y\sim N_n(X\beta,\sigma^2I_n)\), where \(Y\) is an \(n\)-vector, \(X\) is a known \(n\times p\) matrix of rank \(p\), \(\beta\) is a \(p\)-vector, and \(\sigma^2>0\). Suppose the hypothesis is \(H_0:R\beta=r\), where \(R\) has rank \(q\) and has \(q\) rows. The model’s residual degrees of freedom are \(\nu=n-p\), which we assume is positive. The restriction is testable because \(X\) has full column rank.

Definition: For a specified parameter value \(\beta\) satisfying the alternative \(R\beta\ne r\), the exact power of a level-\(\alpha\) test is its rejection probability under that parameter value. For the upper-tail general linear hypothesis F test, power depends on \(\beta\), \(\sigma^2\), \(X\), \(R\), \(r\), \(\alpha\), and the degrees of freedom.

Let \(\hat{\beta}\) be the least-squares estimator, let \(\mathrm{SSE}\) be the residual sum of squares from the full model, and write \(A=R(X'X)^{-1}R'\). This matrix is positive definite because \(R\) has full row rank. The test statistic and the alternative’s noncentrality parameter are

F=\frac{(R\hat{\beta}-r)'A^{-1}(R\hat{\beta}-r)/q}{\mathrm{SSE}/\nu}, \qquad \lambda=\frac{(R\beta-r)'A^{-1}(R\beta-r)}{\sigma^2}.

The earlier material on noncentral F distributions for general linear hypotheses establishes that under \(H_0\), \(F\sim F_{q,\nu}\), while under a specified alternative, \(F\sim F_{q,\nu}(\lambda)\), a noncentral F distribution with \(q\) numerator degrees of freedom, \(\nu\) denominator degrees of freedom, and noncentrality parameter \(\lambda\). In particular, the alternative changes the distribution through \(\lambda\); the degrees of freedom remain those of the test and full model.

The Exact Power Calculation

Let \(c_\alpha\) be the upper-tail critical value from the central F distribution, defined by \(P(F_{q,\nu}>c_\alpha)=\alpha\) when \(F_{q,\nu}\) is central. The test rejects when \(F>c_\alpha\). Its exact power at the specified alternative is therefore the upper-tail probability of the corresponding noncentral F distribution.

Theorem: Under the full-rank normal linear model, for \(0<\alpha<1\), a rank-\(q\) hypothesis, and \(\nu=n-p>0\), the upper-tail F test has exact power

\pi(\beta,\sigma^2) =P_{\beta,\sigma^2}(F>c_\alpha) =1-F_{q,\nu,\lambda}(c_\alpha), \qquad \lambda=\frac{(R\beta-r)'[R(X'X)^{-1}R']^{-1}(R\beta-r)}{\sigma^2},

where \(F_{q,\nu,\lambda}(x)\) denotes the cumulative distribution function of the noncentral F distribution.

To see why the probability gives the test’s power, under the specified alternative the statistic has distribution \(F_{q,\nu}(\lambda)\). The test’s rejection event is \(F>c_\alpha\), so its probability is one minus the noncentral F cumulative probability at \(c_\alpha\). At the null, \(\lambda=0\), the noncentral F distribution reduces to the central F distribution, and the same calculation gives \(1-F_{q,\nu}(c_\alpha)=\alpha\). Thus the calculation also checks the test’s size.

A useful evaluation identity follows by conditioning on a Poisson random variable. If \(K\sim\operatorname{Poisson}(\lambda/2)\), a noncentral F variable can be represented as a central F variable with numerator degrees of freedom \(q+2K\) and denominator degrees of freedom \(\nu\), conditional on \(K\). Consequently,

\pi(\beta,\sigma^2) =\sum_{k=0}^{\infty} e^{-\lambda/2}\frac{(\lambda/2)^k}{k!} \left\{1-F_{q+2k,\nu}(c_\alpha)\right\}.

This is an exact series, not a large-sample approximation. Each term is a Poisson probability multiplied by a central F upper-tail probability. For numerical work, terms can be accumulated until the remaining Poisson probability is sufficiently small; since each central tail is between zero and one, the omitted power contribution is bounded by that remaining probability.

Worked Power Calculations

Worked Example: A Two-Restriction Test with Two Residual Degrees of Freedom

Suppose \(n=8\), \(p=6\), \(X'X=I_6\), and \(\sigma^2=1\). Test \(H_0:\beta_1=\beta_2=0\), so \(R\) selects the first two coordinates, \(r=(0,0)'\), and \(q=2\). Under the specified alternative, let \(\beta=(1,1,0,0,0,0)'\).

The denominator degrees of freedom are \(\nu=8-6=2\). Also, \(A=R I_6 R'=I_2\), so \(\lambda=(1^2+1^2)/1=2\). For \(F_{2,2}\), the central upper tail is \(1/(1+x)\), so setting it equal to \(0.05\) gives \(c_{0.05}=19\). The exact power is therefore

\pi=\sum_{k=0}^{\infty}e^{-1}\frac{1}{k!} \left[ 1-\left\{\frac{19(k+1)}{19(k+1)+1}\right\}^{k+1} \right].

For example, the first conditional tails are \(0.05\) for \(k=0\), \(1-(38/39)^2=77/1521\approx0.050625\) for \(k=1\), and \(1-(57/58)^3=9919/195112\approx0.050838\) for \(k=2\). Weighting these and subsequent terms by \(e^{-1}/k!\) gives power approximately \(0.05046\). It is only slightly above \(0.05\): with just two residual degrees of freedom, the estimated error variance is highly variable, limiting the test’s ability to detect this alternative.

Worked Example: A Larger Alternative in the Same Test

Keep the design, restrictions, variance, and significance level from the previous calculation, but take \(\beta=(2,2,0,0,0,0)'\). The restriction difference is now \((2,2)'\), so \(\lambda=(2^2+2^2)/1=8\). The critical value remains \(19\), since the null distribution and test have not changed. The exact power is

\pi=\sum_{k=0}^{\infty}e^{-4}\frac{4^k}{k!} \left[ 1-\left\{\frac{19(k+1)}{19(k+1)+1}\right\}^{k+1} \right].

The first three Poisson weights are \(e^{-4}\), \(4e^{-4}\), and \(8e^{-4}\). Their corresponding conditional tails are \(0.05\), \(77/1521\), and \(9919/195112\), respectively. For example, the first three contributions to power are \(e^{-4}(0.05)\), \(4e^{-4}(77/1521)\), and \(8e^{-4}(9919/195112)\). Continue the same calculation for \(k=3,4,\ldots\); truncating after any \(k=K\) leaves an error no greater than \(P(K_{\mathrm{Pois}}>K)\), where \(K_{\mathrm{Pois}}\sim\operatorname{Poisson}(4)\). The full series is the exact power, and its larger noncentrality reflects the greater departure from the null.

Worked Example: Changing the Denominator Degrees of Freedom

Now use \(n=16\), \(p=6\), \(X'X=I_6\), and the same two restrictions \(R\beta=0\). Let \(\sigma^2=1\) and \(\beta=(1,1,0,0,0,0)'\), so \(q=2\) and \(\lambda=2\), as before. The residual degrees of freedom are now \(\nu=10\). At \(\alpha=0.05\), the \(F_{2,10}\) critical value is approximately \(4.103\): the central upper tail is \((1+2x/10)^{-5}\), and solving \((1+2x/10)^{-5}=0.05\) gives

c_{0.05}=\frac{10}{2}\left(0.05^{-1/5}-1\right)\approx4.103.

The power is \(1-F_{2,10,2}(4.103)\). Equivalently, the exact Poisson-mixture calculation is

\sum_{k=0}^{\infty}e^{-1}\frac{1}{k!} \left\{1-F_{2+2k,10}(4.103)\right\}.

This example changes the design’s residual degrees of freedom, not the alternative’s noncentrality parameter or the number of tested restrictions. The critical value and the central F tails in the mixture both change. More residual degrees of freedom generally make the error-variance estimate more stable, but the exact power comparison must use the new critical value and distribution; it cannot be determined by comparing critical values alone.

What the Degrees of Freedom Tell You

The numerator degrees of freedom \(q\) count the independent restrictions tested jointly. They determine the numerator distribution and the central critical value, and they also enter the scale of the test statistic through division by \(q\). The denominator degrees of freedom \(\nu=n-p\) describe the residual information used to estimate the error variance. They determine both the denominator distribution and the critical value.

Neither degree of freedom has a simple, universal effect on power when changed in isolation. Changing \(q\) usually means changing the hypothesis being tested, and the alternative’s noncentrality parameter can change as well. Changing the design or model dimension can alter \(\nu\), the covariance of \(\hat{\beta}\), and thus \(\lambda\). A valid comparison specifies the alternatives and model quantities being held fixed, then recalculates the critical value and noncentral F tail for each test.

In practice, an exact power calculation requires a clearly specified alternative, not merely the statement that the null is false. For a multi-parameter test, different alternatives can produce different values of \(\lambda\), even with the same \(q\) and \(\nu\). Power is therefore a property of the test together with the specified model, variance, and alternative.

Common Mistakes

Using the central F distribution under an alternative. The central distribution gives the null reference and critical value. Power uses the noncentral F upper tail at the alternative’s \(\lambda\).
Using the wrong numerator degrees of freedom. For \(q\) independent restrictions, use \(q\), not the total number of model coefficients.
Forgetting the error variance in \(\lambda\). The squared departure from the restrictions is scaled by \(\sigma^2\). Holding the departure fixed, increasing error variance reduces \(\lambda\).
Calling a critical value a power calculation. The critical value is chosen under the null. Power additionally requires the alternative distribution and its noncentrality parameter.
Assuming that more degrees of freedom automatically mean more power. The critical value, distribution, model, and possibly \(\lambda\) all matter. Recompute the complete tail probability under the specified comparison.

Key takeaway: Specify the model and alternative, calculate \(q\), \(\nu\), and \(\lambda\), find the central F critical value at the chosen significance level, and evaluate the noncentral F upper tail. That tail probability is the exact finite-sample power.

Check Your Understanding

For each question, identify which part of the exact power calculation is involved.

In a full-rank normal linear model with \(n=30\), \(p=8\), and five independent restrictions, what are the numerator and denominator degrees of freedom?
Why does the critical value come from a central F distribution even though power is evaluated using a noncentral F distribution?
For a fixed design and restriction, what happens to \(\lambda\) if the restriction departure stays fixed but \(\sigma^2\) is multiplied by four?
How can the Poisson mixture representation be used to bound the numerical error from truncating a power calculation?
Why is comparing two numerator degrees of freedom without specifying the corresponding alternatives potentially misleading?