How Significance Level Changes Power

Why the Significance Level Changes Power

A hypothesis test needs a rule for deciding when sample evidence is sufficiently inconsistent with the null hypothesis to reject it. The significance level, \(\alpha\), sets how readily that rule leads to rejection. A larger \(\alpha\) makes rejection easier. This can help a test detect a real effect, but it also raises the chance of rejecting the null hypothesis when it is true.

Power describes a test’s ability to reject the null when a particular alternative is true. It depends on which alternative is considered: an effect close to the null is generally harder to detect than a more distant effect. In this tutorial, we compare power across significance levels while holding the alternative, sample size, and variability fixed. The gain in power is therefore not free: it comes with a higher Type I error risk.

Definition: For a specified alternative value or model, power is the probability that the test rejects \(H_0\) when that alternative is true. The significance level \(\alpha\) is the probability, under \(H_0\), of rejecting \(H_0\) for a test calibrated at level \(\alpha\). For a continuous test calibrated exactly, this probability equals \(\alpha\).

The significance level is not the probability that \(H_0\) is true, and power is not the probability that a particular study will succeed. They are probabilities describing how a test procedure behaves under specified conditions. A test’s power must be tied to a particular alternative, along with assumptions about the data and the design.

A Normal-Mean Test at Different Alpha Levels

Consider a one-sided test of a population mean with a known population standard deviation. Suppose the observations are independent and come from a normal population. Then the sample mean has a normal distribution with standard error \(SE=\sigma/\sqrt{n}\), where \(\sigma\) is the population standard deviation and \(n\) is the sample size. Under \(H_0:\mu=\mu_0\), the statistic \(Z=(\bar{x}-\mu_0)/SE\) follows the standard normal distribution.

For \(H_a:\mu>\mu_0\), the test rejects when \(Z>z_{1-\alpha}\), where \(z_{1-\alpha}\) is the standard normal quantile with cumulative probability \(1-\alpha\). At a specified alternative \(\mu=\mu_0+\delta\), where \(\delta>0\), the statistic has a normal distribution with standard deviation 1 and mean \(\lambda=\delta/SE\). Writing \(\Phi\) for the standard normal cumulative distribution function, power is the probability that this shifted statistic falls in the rejection region:

\text{Power}(\delta;\alpha) = P(Z>z_{1-\alpha}\mid \mu=\mu_0+\delta) = 1-\Phi(z_{1-\alpha}-\lambda) = \Phi(\lambda-z_{1-\alpha}).

Holding \(\delta\), \(SE\), and therefore \(\lambda\) fixed, increasing \(\alpha\) lowers the critical value \(z_{1-\alpha}\). The test then rejects for a broader range of observed statistics. More results fall in that rejection region under the specified alternative, so power increases. The same broader region also includes more outcomes that can occur under the null, raising the Type I error probability.

Worked Example: A One-Sided Test at Three Significance Levels

Suppose a hypothetical process produces independent normal measurements with known population standard deviation \(\sigma=10\) units. A sample of \(n=25\) measurements is used to test \(H_0:\mu=100\) against \(H_a:\mu>100\). Find the power at the specified alternative \(\mu=104\) for \(\alpha=0.01,\ 0.05,\) and \(0.10\).

Check the model. The measurements are assumed independent and normally distributed, and \(\sigma=10\) is treated as known. Thus the sample mean has the stated normal distribution, with \(SE=10/\sqrt{25}=2\) units. The alternative is fixed at 104 units, so the shift relative to the null is \(\delta=4\) units and \(\lambda=4/2=2\).

For a one-sided test, the power formula is \(\Phi(\lambda-z_{1-\alpha})\). Using \(z_{0.99}=2.326\), \(z_{0.95}=1.645\), and \(z_{0.90}=1.282\):

\begin{aligned} \alpha=0.01:\quad &\text{Power}=\Phi(2-2.326)=\Phi(-0.326)\approx 0.372,\\ \alpha=0.05:\quad &\text{Power}=\Phi(2-1.645)=\Phi(0.355)\approx 0.639,\\ \alpha=0.10:\quad &\text{Power}=\Phi(2-1.282)=\Phi(0.718)\approx 0.764. \end{aligned}

At this fixed alternative, the test has about 37.2% power at the 1% significance level and about 63.9% power at the 5% level. Raising \(\alpha\) to 10% raises power to about 76.4%. The corresponding Type I error probabilities, if the null model holds and the test is correctly calibrated, are 1%, 5%, and 10%. These are properties of the test procedure, not probabilities that the null hypothesis is true.

Two-Sided Tests: Power in Both Tails

For a two-sided test, evidence against \(H_0\) can fall in either tail of the null distribution. Under the same known-\(\sigma\), independent normal model, the test of \(H_0:\mu=\mu_0\) against \(H_a:\mu\ne\mu_0\) rejects when \(|Z|>z_{1-\alpha/2}\). At an alternative with standardized shift \(\lambda=(\mu-\mu_0)/SE\), the statistic is normally distributed with mean \(\lambda\) and standard deviation 1. Power is the probability of landing in either rejection tail:

\begin{aligned} \text{Power}(\lambda;\alpha) &=P(Z<-z_{1-\alpha/2}\mid \lambda) +P(Z>z_{1-\alpha/2}\mid \lambda)\\ &=\Phi(-z_{1-\alpha/2}-\lambda) +1-\Phi(z_{1-\alpha/2}-\lambda). \end{aligned}

As \(\alpha\) increases, \(z_{1-\alpha/2}\) decreases, making the two-sided rejection region wider. Consequently, more outcomes under a fixed nonzero alternative are included in that region. Unlike a one-sided test, a two-sided test allocates its Type I error risk across both tails; its critical value must therefore use \(\alpha/2\) in each tail.

Worked Example: Two-Sided Power at Three Alpha Levels

In a hypothetical setting, independent measurements are normally distributed with known \(\sigma=10\) units. A sample of \(n=64\) is used to test \(H_0:\mu=50\) against \(H_a:\mu\ne50\). Calculate power when the specified alternative is \(\mu=53\), at \(\alpha=0.01,\ 0.05,\) and \(0.10\).

Check the model. The normal population, independent measurements, and known population standard deviation justify the normal test model. The standard error is \(SE=10/\sqrt{64}=1.25\) units. At \(\mu=53\), \(\lambda=(53-50)/1.25=2.4\). For a two-sided test, the critical values use \(z_{1-\alpha/2}\).

Using critical values 2.576, 1.960, and 1.645, respectively, substitute into the two-tail power formula:

\begin{aligned} \alpha=0.01:\quad &\Phi(-2.576-2.4)+1-\Phi(2.576-2.4) \approx 0.0000+1-\Phi(0.176)\approx 0.430,\\ \alpha=0.05:\quad &\Phi(-1.960-2.4)+1-\Phi(1.960-2.4) \approx 0.0000+1-\Phi(-0.440)\approx 0.670,\\ \alpha=0.10:\quad &\Phi(-1.645-2.4)+1-\Phi(1.645-2.4) \approx 0.0000+1-\Phi(-0.755)\approx 0.775. \end{aligned}

The lower-tail probabilities are very small for this positive alternative, but they are included in the formula. At the specified mean of 53 units, power rises from about 43.0% to 67.0% to 77.5% as \(\alpha\) increases. Under the null, the corresponding overall Type I error probabilities are 1%, 5%, and 10%, split between the two tails.

Why the Trade-Off Must Occur

The power increase can be justified without calculating a normal probability. A test rejects when its test statistic belongs to a rejection region. When the significance level increases, the critical boundary moves so that the rejection region expands. For example, in a one-sided upper-tail test, \(z_{1-\alpha}\) decreases as \(\alpha\) increases. Thus if \(\alpha_1<\alpha_2\), then \(z_{1-\alpha_1}>z_{1-\alpha_2}\), and:

\{Z>z_{1-\alpha_1}\}\subseteq\{Z>z_{1-\alpha_2}\}.

For any fixed alternative distribution, the probability of the larger rejection region cannot be smaller than the probability of the contained region. Therefore power at \(\alpha_2\) is at least power at \(\alpha_1\). Under a continuous distribution with positive probability between the two boundaries, the increase is strict. Under the null distribution, the expanded region also has a larger probability; for an exactly calibrated test, that probability is the larger significance level itself. The same reasoning applies to two-sided tests, whose rejection regions expand as the absolute critical value decreases.

Worked Example: A Paired Study and a Practical Choice

A hypothetical study measures each of 36 participants before and after a program. The outcome is the within-participant change, defined so that a positive value indicates an increase. Suppose these changes are independent across participants, normally distributed, and have known population standard deviation \(\sigma=3\) units. A one-sided test assesses \(H_0:\mu_d=0\) against \(H_a:\mu_d>0\), where \(\mu_d\) is the population mean change. Compare power at \(\mu_d=1.5\) units for \(\alpha=0.01,\ 0.05,\) and \(0.10\).

Check the model. The analysis uses one change score per participant, so the sample size is 36 independent pairs, not 72 independent observations. Under the stated normal-change model and known standard deviation, \(SE=3/\sqrt{36}=0.5\) units. At the specified alternative, \(\lambda=1.5/0.5=3\).

This is an upper-tail test, so power is \(\Phi(\lambda-z_{1-\alpha})\). Substituting the critical values gives:

\begin{aligned} \alpha=0.01:\quad &\Phi(3-2.326)=\Phi(0.674)\approx 0.750,\\ \alpha=0.05:\quad &\Phi(3-1.645)=\Phi(1.355)\approx 0.912,\\ \alpha=0.10:\quad &\Phi(3-1.282)=\Phi(1.718)\approx 0.957. \end{aligned}

For a true mean increase of 1.5 units, moving from \(\alpha=0.01\) to \(\alpha=0.05\) raises power from about 75.0% to 91.2%. But it also changes the null-based Type I error probability from 1% to 5%. Whether that trade-off is appropriate depends on the consequences of false positives and missed effects; the arithmetic alone cannot decide the scientific choice.

Choosing and Reporting Alpha Carefully

A significance level should be chosen before examining the study’s results, with the relative costs of false positives and false negatives in mind. A smaller \(\alpha\) makes the test more demanding before it rejects, controlling the Type I error probability at a lower level but reducing power at any fixed alternative. A larger \(\alpha\) does the reverse. Neither choice changes the underlying effect, the sample size, or the variability in these comparisons; it changes the decision threshold.

When reporting a planned power calculation, state the significance level, test direction, sample size, variability assumption, and specific alternative. A statement such as “power is 80%” is incomplete unless the alternative and design assumptions are clear. Power computed for one effect does not automatically describe the chance of detecting every possible effect.

Common mistake: “Raising alpha improves power, so raising it after seeing a nonsignificant result is a reasonable way to detect the effect.”
Correction: Raising alpha does increase power for a fixed alternative, but it also raises the Type I error probability. Changing the threshold after seeing the data means the test no longer follows the originally specified decision rule and its stated error-rate guarantee cannot be used without accounting for that change.

Key takeaway: With the alternative, sample size, and variability fixed, increasing \(\alpha\) expands the rejection region and increases power. It also increases the chance of a Type I error under \(H_0\). Choosing \(\alpha\) is a decision about that trade-off, not a way to improve detection without cost.

Check Your Understanding

Use the definitions and normal-test formulas above to answer these questions.

For a one-sided upper-tail test, why does increasing \(\alpha\) lower the critical value?
At a fixed alternative, what happens to power when the sample size and variability stay fixed but \(\alpha\) increases?
In a two-sided test with \(\alpha=0.06\), what significance probability is allocated to each tail under a symmetric null distribution?
A researcher says that power is 90% without naming an alternative effect. What important information is missing?
Why is it misleading to increase the chosen significance level only after a test has produced a nonsignificant result?