Exact Power for One- and Two-Sided t Tests

From a Test Statistic to Power

A t test rejects when its observed statistic falls in a specified tail or tails of a central t distribution. To calculate power, the probability of rejection when a particular alternative is true, use the distribution of that same statistic under the alternative. In the normal-theory setting, this is a noncentral t distribution. The calculation is finite-sample: it uses the actual residual degrees of freedom rather than replacing the t distribution with a normal approximation.

The relevant alternative must be specified quantitatively. For a contrast, for example, power depends on how far its true value is from the null value, measured relative to the true standard deviation of its estimate. The direction of the difference matters for a one-sided test; its magnitude matters for a two-sided test.

For a normal-theory test of a contrast, suppose the estimate \(\hat\theta\) has mean \(\theta\) and variance \(\sigma^2 q\), with \(q>0\). The standard error is estimated using an independent residual variance estimate \(MSE\), with \(\nu\) residual degrees of freedom and \(\nu MSE/\sigma^2\sim\chi^2_\nu\). Testing \(H_0:\theta=\theta_0\) uses \(T=(\hat\theta-\theta_0)/\sqrt{MSEq}\). Under a specified alternative \(\theta\), its noncentrality parameter is \(\delta=(\theta-\theta_0)/(\sigma\sqrt q)\). These distributional conditions hold, for example, for a normal linear model when the contrast estimate is normal and independent of the residual sum of squares.

Let \(t_{p,\nu}\) denote the \(p\)-quantile of the central t distribution with \(\nu\) degrees of freedom. A test at level \(\alpha\) uses these central-t quantiles to set its rejection region. Once that region is fixed, its probability under the alternative is computed using \(F_{\nu,\delta}\).

Exact Power Formulas

Derivation. Under the specified alternative, \(T\) has cumulative distribution function \(F_{\nu,\delta}\). For the upper-tailed test, the rejection event is \(T>c\), where \(c=t_{1-\alpha,\nu}\), so its probability is \(1-P(T\le c)=1-F_{\nu,\delta}(c)\). For the lower-tailed test the event is \(T<-c\), giving probability \(F_{\nu,\delta}(-c)\). For the two-sided test, the events \(T<-c_2\) and \(T>c_2\), with \(c_2=t_{1-\alpha/2,\nu}\), are disjoint. Adding their probabilities gives the stated expression. The quantiles come from the null distribution because they determine the test; the cdf in the power calculation comes from the alternative distribution. \(\square\)

The noncentral t cdf can also be represented as an expectation that makes the role of variance estimation explicit. Since \(T=(Z+\delta)/\sqrt{U/\nu}\), conditioning on \(U\) gives

where \(\Phi\) is the standard normal cdf and \(f_{\chi^2_\nu}\) is the chi-square density with \(\nu\) degrees of freedom. Indeed, for a fixed value \(U=u\), the event \(T\le c\) is equivalent to \(Z\le c\sqrt{u/\nu}-\delta\). Its conditional probability is the integrand’s normal cdf factor. Integrating that conditional probability against the density of \(U\) gives the displayed formula. This representation also explains why replacing the random denominator with a constant generally does not give exact finite-sample power.

Alternative Direction and Degrees of Freedom

For a lower-tailed test, the alternative should put the noncentrality in the negative direction. For instance, in the contrast setup above, if \(\theta-\theta_0<0\), then \(\delta<0\). The lower-tail formula \(F_{\nu,\delta}(-t_{1-\alpha,\nu})\) then gives the probability of rejection in the direction specified by the test. A positive alternative paired with a lower-tailed rejection rule can have low power; changing the sign of \(\delta\) without changing the rule would answer a different question.

Degrees of freedom affect power in two places: they determine the critical value and the distribution of the random denominator. The critical value is larger for many small-\(\nu\) tests at the same nominal level, while the denominator is also more variable. Consequently, comparing powers across degrees of freedom requires using each degree of freedom’s own critical value and noncentral t distribution. There is no valid exact comparison obtained by changing only the critical value or only the cdf.

Why the Noncentrality Parameter Matters

The parameter \(\delta=(\theta-\theta_0)/(\sigma\sqrt q)\) is the alternative’s standardized distance from the null. It combines the specified effect with the true variability and the contrast’s design variance factor. For a one-sample mean, \(q=1/n\), so \(\delta=\sqrt n(\mu-\mu_0)/\sigma\). A larger sample size increases the noncentrality when the effect and \(\sigma\) stay fixed. In a contrast test, \(q\) depends on the contrast and design, so two alternatives with the same raw difference can have different power.

For a two-sided test, power is symmetric in \(\delta\): changing the sign of the alternative exchanges the lower and upper rejection tails. More generally, for fixed \(\nu\) and critical value \(c>0\), two-sided power increases as \(|\delta|\) increases. To see this, condition on \(U\) and write \(a=c\sqrt{U/\nu}>0\). Given \(U\), the probability of rejection is \(\Phi(-a-\delta)+1-\Phi(a-\delta)\). For \(\delta\ge0\), its derivative with respect to \(\delta\) is \(\phi(a-\delta)-\phi(a+\delta)\ge0\), since \(|a-\delta|\le a+\delta\) and the standard normal density decreases as the absolute value of its argument increases. Averaging over \(U\) preserves this monotonicity. Symmetry follows by replacing \(Z\) with \(-Z\). \(\square\)

Common Mistakes

• Using the central t distribution under the alternative. The central t distribution supplies the null critical value. Under a nonzero alternative, the statistic has a noncentral t distribution, so power must use \(F_{\nu,\delta}\).
• Using the observed standard error to define the noncentrality. The noncentrality uses the true standard deviation \(\sigma\sqrt q\), not the random observed value \(\sqrt{MSEq}\).
• Counting only one tail in a two-sided test. A two-sided test rejects in either tail, so both probabilities must be added.
• Ignoring the direction of a one-sided alternative. The sign of \(\delta\) must agree with the alternative and the rejection tail.
• Changing degrees of freedom but retaining the old critical value. The test’s critical value and its noncentral t distribution must both use the correct residual degrees of freedom.