Power for Two-Sample Z Tests

Why Power Depends on the Difference Between Two Means

A two-sample Z test compares the means of two independent populations when their variances are known. Its power is not a single number: it depends on the true difference between the population means. A test designed to detect a substantial difference will generally have less power against a small difference, because the sample means are then less likely to cross the rejection threshold.

For planning or interpretation, the key steps are to identify the test's rejection region, find the distribution of the test statistic under the alternative of interest, and calculate the probability that the statistic falls in that region. The sample sizes of the two groups enter separately through the variance of the difference in sample means.

Model and assumptions: The samples are independent, with \(X_{11},\ldots,X_{1n_1}\) drawn from \(N(\mu_1,\sigma_1^2)\) and \(X_{21},\ldots,X_{2n_2}\) drawn from \(N(\mu_2,\sigma_2^2)\). The population variances \(\sigma_1^2\) and \(\sigma_2^2\) are known, the sample sizes are fixed, and observations within each sample are independent and identically distributed. Define \(\delta=\mu_1-\mu_2\). The exact normal calculations below require the normal-population assumption; without it, they are generally approximations unless a suitable large-sample justification applies.

The Statistic and Its Rejection Region

Let \(\bar X_1\) and \(\bar X_2\) be the two sample means. Independence gives the variance of their difference by adding the two variances. The standard error is therefore

SE=\sqrt{\frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2}}, \qquad Z=\frac{\bar X_1-\bar X_2}{SE}.

Under the null hypothesis \(H_0:\mu_1-\mu_2=0\), \(Z\sim N(0,1)\). The critical values are chosen to give the test its specified significance level \(\alpha\). Write \(\Phi(z)=P(W\le z)\) for the standard normal cumulative distribution function when \(W\sim N(0,1)\), and let \(z_q\) satisfy \(\Phi(z_q)=q\).

Alternative	Rejection region at level \(\alpha\)
\(H_1:\delta>0\)	\(Z>z_{1-\alpha}\)
\(H_1:\delta<0\)	\(Z<-z_{1-\alpha}\)
\(H_1:\delta\ne 0\)	\(Z<-z_{1-\alpha/2}\) or \(Z>z_{1-\alpha/2}\)

The rejection region is set using the null distribution and the chosen \(\alpha\). It does not move when evaluating power at an alternative. Power is the probability of rejecting the null when the alternative's parameter values are true; equivalently, it is one minus the Type II error probability at those values.

Distribution Under an Alternative and Power Formulas

Under a specified alternative with difference \(\delta\), the mean difference \(\bar X_1-\bar X_2\) has mean \(\delta\) and variance \(SE^2\). Consequently, the same test statistic used for the rejection rule has a shifted distribution under the alternative:

Z\sim N(\lambda,1), \qquad \lambda=\frac{\delta}{SE}.

The quantity \(\lambda\) is the alternative mean difference measured in standard-error units. For a positive alternative difference, it shifts the distribution to the right; a negative difference shifts it to the left. To obtain power, calculate the probability of the null rejection region under this shifted distribution.

Power formulas: For an upper-tailed test, \(\operatorname{Power}(\delta)=1-\Phi(z_{1-\alpha}-\lambda)\). For a lower-tailed test, \(\operatorname{Power}(\delta)=\Phi(-z_{1-\alpha}-\lambda)\). For a two-sided test, \(\operatorname{Power}(\delta)=\Phi(-z_{1-\alpha/2}-\lambda)+1-\Phi(z_{1-\alpha/2}-\lambda)\), where \(\lambda=\delta/SE\).

For example, the upper-tailed formula follows directly from the rejection event. Under the alternative, write \(Z=\lambda+W\), where \(W\sim N(0,1)\). Then

P(Z>z_{1-\alpha}) =P(\lambda+W>z_{1-\alpha}) =P(W>z_{1-\alpha}-\lambda) =1-\Phi(z_{1-\alpha}-\lambda).

For a two-sided test, the rejection events are disjoint: \(Z<-z_{1-\alpha/2}\) and \(Z>z_{1-\alpha/2}\). Evaluating each event under \(Z\sim N(\lambda,1)\) gives the sum of the two tail probabilities in the formula. The same reasoning gives the lower-tailed expression.

Worked Power Calculations

Worked Example: Power for an Upper-Tailed Test

A hypothetical study compares mean processing times for two independent normal populations. The known standard deviations are \(\sigma_1=9\) and \(\sigma_2=12\) seconds, with sample sizes \(n_1=36\) and \(n_2=64\). Test \(H_0:\delta=0\) against \(H_1:\delta>0\) at \(\alpha=0.05\). Find the power when the true difference is \(\delta=4\) seconds.

Method. Use the upper-tailed Z rejection region and evaluate it under the specified alternative. The two sample sizes and variances must be used in their respective terms:

SE=\sqrt{\frac{9^2}{36}+\frac{12^2}{64}} =\sqrt{\frac{81}{36}+\frac{144}{64}} =\sqrt{2.25+2.25} =\sqrt{4.5} =2.1213.

The alternative shift is \(\lambda=4/\sqrt{4.5}=1.8856\). At level \(0.05\), \(z_{0.95}=1.6449\), so reject \(H_0\) when \(Z>1.6449\). Thus

\begin{aligned} \operatorname{Power}(4) &=P(Z>1.6449\mid \delta=4)\\ &=1-\Phi(1.6449-1.8856)\\ &=1-\Phi(-0.2408)\\ &=\Phi(0.2408)\\ &\approx 0.5951. \end{aligned}

The power is 0.5951, or about 59.51%. A common error is to report \(\Phi(-0.2408)\approx0.4049\) as the power. That is the probability of falling below the upper-tail cutoff, not the probability of rejecting; the rejection probability is its complement. The result is consistent with a right-shifted alternative distribution, for which rejection is more likely than non-rejection.

Worked Example: Power for a Two-Sided Test

A hypothetical comparison of two independent normal measurement populations has known standard deviations \(\sigma_1=10\) and \(\sigma_2=14\), and sample sizes \(n_1=25\) and \(n_2=49\). A two-sided test of \(H_0:\delta=0\) has \(\alpha=0.05\). Find its power when the true difference is \(\delta=7\).

Method. Find the standard error and alternative shift, then add the probabilities of both rejection tails. First,

SE=\sqrt{\frac{10^2}{25}+\frac{14^2}{49}} =\sqrt{4+4} =\sqrt{8} =2.8284, \qquad \lambda=\frac{7}{\sqrt{8}}=2.4749.

The two-sided critical value is \(z_{0.975}=1.9600\), so the rejection region is \(Z<-1.9600\) or \(Z>1.9600\). Under the alternative, \(Z\sim N(2.4749,1)\), giving

\begin{aligned} \operatorname{Power}(7) &=\Phi(-1.9600-2.4749)+1-\Phi(1.9600-2.4749)\\ &=\Phi(-4.4349)+1-\Phi(-0.5149)\\ &\approx 0.0000+0.6967\\ &=0.6967. \end{aligned}

The power is 0.6967. The lower-tail contribution rounds to 0.0000 to four decimals, but it is included in the formula. A frequent mistake is to compute only the upper tail without first recognizing that the test is two-sided. Here the omitted tail is tiny, but for other alternatives or sample sizes it need not be.

Worked Example: Power for a Lower-Tailed Test

A hypothetical experiment compares independent normal populations with known standard deviations \(\sigma_1=5\) and \(\sigma_2=10\). The sample sizes are \(n_1=25\) and \(n_2=100\). Test \(H_0:\delta=0\) against \(H_1:\delta<0\) at \(\alpha=0.10\). Calculate the power when \(\delta=-3\).

Method. The lower-tailed rejection boundary is the negative critical value. Calculate the standard error using the separate variances and sample sizes:

SE=\sqrt{\frac{5^2}{25}+\frac{10^2}{100}} =\sqrt{1+1} =\sqrt{2} =1.4142, \qquad \lambda=\frac{-3}{\sqrt{2}}=-2.1213.

Since \(z_{0.90}=1.2816\), reject when \(Z<-1.2816\). Under the alternative, the probability of rejection is

\begin{aligned} \operatorname{Power}(-3) &=P(Z<-1.2816\mid\delta=-3)\\ &=\Phi(-1.2816-(-2.1213))\\ &=\Phi(0.8398)\\ &\approx 0.7995. \end{aligned}

The power is 0.7995. The sign of the alternative matters: for a negative \(\delta\), the distribution shifts toward the lower-tail rejection region. A common error is to use a positive critical boundary or to subtract the negative shift as if it were positive; keeping the event \(Z<-1.2816\) visible prevents both mistakes.

Interpreting Power and Avoiding Errors

For fixed known variances and sample sizes, increasing \(|\delta|\) increases \(|\lambda|\), moving the alternative distribution farther from the null center. Power therefore rises in the direction of the chosen one-sided alternative. Increasing either sample size reduces its contribution \(\sigma_j^2/n_j\) to the standard error, which generally makes a fixed nonzero difference easier to detect. In a two-sided test, power can also be contributed by the tail opposite the direction of the true difference, although that contribution may be very small.

The formulas use the variances of the two sample means, not a pooled variance. In particular, the standard error is \(\sqrt{\sigma_1^2/n_1+\sigma_2^2/n_2}\); it is not obtained by adding the standard deviations, nor by treating \(n_1\) and \(n_2\) as interchangeable when their variances differ. The known-variance assumption is important: if the variances are estimated from the samples, this is not the known-variance Z procedure and these standard normal power formulas do not apply as written.

Power is always tied to a specified alternative value. For a one-sided test, the direction in the alternative must agree with the rejection tail. For a two-sided test, both rejection tails must be counted. Keep the rejection event fixed from the null test, shift the statistic's distribution to the alternative mean \(\lambda\), and then evaluate the probability.

Key takeaway: For two independent normal samples with known variances, calculate \(SE=\sqrt{\sigma_1^2/n_1+\sigma_2^2/n_2}\), set \(\lambda=\delta/SE\), and evaluate the test's null rejection region under \(N(\lambda,1)\). Power is that probability—not the probability of failing to reject.

Check Your Understanding

For each problem, use the stated known-variance normal model and round the final power to four decimals.

A one-sided upper test has \(\alpha=0.05\), \(\sigma_1=6\), \(\sigma_2=8\), \(n_1=16\), \(n_2=25\), and true difference \(\delta=3\). What is its power? (Answer: 0.3909)
A two-sided test has \(\alpha=0.10\), \(\sigma_1=4\), \(\sigma_2=5\), \(n_1=16\), \(n_2=25\), and true difference \(\delta=-2\). What is its power, including both tails? (Answer: 0.4099)
A lower-tailed test has \(\alpha=0.05\), \(\sigma_1=3\), \(\sigma_2=4\), \(n_1=9\), \(n_2=16\), and true difference \(\delta=-2\). Find its power. (Answer: 0.4088)
A one-sided upper test has \(\alpha=0.01\), \(\sigma_1=12\), \(\sigma_2=9\), \(n_1=36\), \(n_2=81\), and true difference \(\delta=5\). Find the standard error and power. (Answer: \(SE=\sqrt{5}=2.2361\); power \(=0.4640\))
A two-sided test has \(\alpha=0.05\), any positive sample sizes, and any known population standard deviations. If the true difference is exactly \(\delta=0\), what is its power under this model? (Answer: 0.0500)

What You'll Learn