Power as the Probability of Detecting an Effect

What Does It Mean for a Test to Have Power?

A hypothesis test gives a decision rule for data: under certain conditions, the rule rejects \(H_0\); otherwise, it does not reject \(H_0\). But before collecting data, we can ask how often that rule would reject the null if a particular alternative condition were actually true. That probability describes the test’s ability to detect an effect of that specified size.

This ability is called power. Power is not a statement about what is probably true after seeing a result. It describes the long-run behavior of a test under a stated alternative: if the same study and decision rule were repeated many times while that alternative condition held, what fraction of the tests would reject \(H_0\)?

Definition: For a specified alternative condition, the power of a hypothesis test is the probability that the test rejects \(H_0\) when that alternative condition is true. If \(\beta\) is the probability of a Type II error under that same condition, then power is \(1-\beta\).

A test’s power depends on the decision rule and on the condition assumed to be true. In particular, it is not enough to say “the test has power 80%” without saying under which alternative that probability is being described. For example, the probability of rejecting \(H_0\) may differ when the true mean change is 1 unit rather than 4 units.

This tutorial treats the test and its decision rule as already specified. The main task is to identify the condition under which power is being evaluated, then interpret the probability of rejection under that condition.

Power and Type II Error Are Complementary

Fix a particular alternative condition. On any one repetition of the study, the test either rejects \(H_0\) or does not reject \(H_0\). Under that fixed condition, these two outcomes are complementary. A Type II error occurs when the test does not reject \(H_0\) even though the specified alternative condition is true.

\text{Power}=P(\text{reject }H_0\mid\text{specified alternative is true})

\beta=P(\text{do not reject }H_0\mid\text{specified alternative is true})

Because rejection and non-rejection exhaust the possible decisions, their probabilities under the same condition add to 1. Therefore:

\text{Power}+\beta=1 \qquad\Longrightarrow\qquad \text{Power}=1-\beta

This identity does not say that a particular study must detect an effect whenever power is high. Power is a probability over repetitions, not a guarantee about a single result. A test with power 0.80 under a specified alternative would still fail to reject \(H_0\) in about 20% of repeated studies under that condition, in the long run.

Worked Examples

Worked Example: Power for a Simple Count Test

Suppose a hypothetical test concerns a spinner landing on a marked sector. Let \(p\) be the probability of landing on that sector on one spin. The null hypothesis is \(H_0:p=0.50\), and the alternative of interest is \(p=0.70\). The spinner is spun 5 times independently. The test rejects \(H_0\) if at least 4 spins land on the marked sector. Find the power under \(p=0.70\).

Model and assumptions. For this calculation, assume each spin has the same probability \(p=0.70\) of landing on the marked sector and that the five spins are independent. Let \(X\) be the number of marked-sector results. Then \(X\) has a binomial distribution with \(n=5\) and success probability \(0.70\). The rejection rule is \(X\geq 4\).

Power is the probability of rejection when the specified alternative is true:

\text{Power}=P(X\geq 4\mid p=0.70)

The test rejects for either exactly 4 or exactly 5 marked-sector results. Using the binomial probabilities:

\begin{aligned} P(X\geq 4\mid p=0.70) &=P(X=4)+P(X=5)\\ &=\binom{5}{4}(0.70)^4(0.30)+(0.70)^5\\ &=5(0.2401)(0.30)+0.16807\\ &=0.36015+0.16807\\ &=0.52822. \end{aligned}

Thus the power is about \(0.5282\), or 52.82%, when the true probability of a marked-sector result is 0.70. Under that same condition, the probability of a Type II error is \(1-0.52822=0.47178\), or 47.18%. These are probabilities for repeated sets of five spins under the stated model, not probabilities that \(p=0.70\) is true.

Worked Example: Estimating Power by Simulation

A hypothetical paired study plans to test whether a treatment produces a positive mean change. The plan uses a one-sided paired \(t\) test with a fixed significance level of \(\alpha=0.05\) and \(n=16\) pairs. Investigators want to describe power when the true mean change is 2.0 points and the standard deviation of paired differences is 3.0 points.

Model and assumptions. Suppose the simulated paired differences are independent draws from a normal distribution with mean 2.0 points and standard deviation 3.0 points. In each simulated study, the same pre-specified test and rejection rule are applied. These assumptions define the conditions under which the simulation estimates power; the simulation does not establish that those conditions hold in a real study.

Imagine that 1,000 simulated studies are generated under this model and the test rejects \(H_0\) in 684 of them. The estimated probability of rejection is the proportion of simulated studies that reject:

\widehat{\text{Power}}=\frac{684}{1000}=0.684

The estimated power is 68.4% for detecting a positive mean change when the true mean change is 2.0 points under the stated model and study plan. The complementary estimated Type II error probability is:

\widehat{\beta}=1-0.684=0.316

So, under those same assumptions, about 31.6% of repeated simulated studies do not reject \(H_0\). The value 0.684 is an estimate based on 1,000 simulations; it may differ slightly from the model’s exact power because the simulation uses a finite number of repetitions.

Worked Example: Interpreting Power After a Study

A hypothetical clinical study has a pre-specified test for a mean reduction in symptom score. Before the study begins, its design is evaluated under a model in which the true mean reduction is 3 points. Under that condition, the probability that the test rejects \(H_0\) is reported as 0.80. The completed study produces a result that does not reject \(H_0\). What does the power statement mean for this study?

Identify the condition. The reported power applies when the true mean reduction is 3 points, with the study design, model assumptions, and decision rule used in the evaluation. It is not a power value for every possible mean reduction.

Interpret the probability. If the true mean reduction were 3 points and the planned study were repeated under those conditions, the test would reject \(H_0\) in about 80% of repetitions. The complementary probability of not rejecting under that same condition is:

\beta=1-0.80=0.20

The observed non-rejection is one possible outcome, even when the true mean reduction is 3 points. The power statement does not mean there is an 80% probability that the true mean reduction is 3 points, nor does it tell us the probability that \(H_0\) is true after seeing the study result.

Why the Alternative Must Be Specified

Power is conditional on a state of the world in which the null hypothesis is not the condition being evaluated. If an alternative hypothesis includes a range of possible values, the probability of rejection can vary across that range. Thus, a power statement usually names a particular value or scenario within the alternative, such as a true mean change of 3 points or a true success probability of 0.70.

The conditional probability notation makes the direction of the statement explicit. In \(P(\text{reject }H_0\mid\text{specified alternative is true})\), the alternative condition is given, and the probability concerns the test’s decision. It is not the reverse probability \(P(\text{specified alternative is true}\mid\text{test rejects }H_0)\). Reversing the conditioning would ask a different question.

This distinction is important when reading a study plan or a test report. A power value describes how well a particular design and test are expected to detect a particular alternative under stated assumptions. It does not by itself measure the practical value of an effect, establish that the model assumptions are accurate, or explain the result of one completed study.

A Short Justification of the Complement Rule

The relationship between power and \(\beta\) follows directly from the possible decisions of the test. Hold the specified alternative condition fixed. Let \(R\) be the event that the test rejects \(H_0\), and let \(R^c\) be the event that it does not reject \(H_0\). These events are mutually exclusive and together include every possible test decision, so:

P(R\mid\text{alternative})+P(R^c\mid\text{alternative})=1

By definition, the first conditional probability is power, and the second is \(\beta\), the Type II error probability under that alternative. Substituting those definitions gives \(\text{Power}+\beta=1\), or \(\text{Power}=1-\beta\). The condition must be the same in both terms; subtracting a Type II error probability for one alternative from 1 would not give power for a different alternative.

Common Mistakes

Calling power the probability that the alternative is true. Power is the probability of rejecting \(H_0\) conditional on a specified alternative being true. It does not assign a probability to that condition itself.
Using “80% power” without naming the condition. Power is evaluated under an alternative condition. State the assumed effect or other relevant parameter value, as well as the study and test to which the value applies.
Assuming high power guarantees a significant result. Even under the specified alternative, a test with power below 1 sometimes does not reject \(H_0\). Power describes the long-run fraction of rejections, not a guaranteed outcome.
Confusing power with the probability of a Type I error. Power is defined under a specified alternative; the significance level describes the test’s Type I error probability under the null condition. They refer to different conditions and different errors.
Reversing a conditional probability. \(P(\text{reject }H_0\mid\text{alternative})\) is not the probability of the alternative given that the test rejected.

Key takeaway: For a specified alternative condition, power is the probability that the test rejects \(H_0\); the Type II error probability under that same condition is its complement. Power describes repeated-test behavior under the stated condition, not the probability that a hypothesis is true.

Check Your Understanding

Answer each question by naming the condition that the probability refers to.

A test has power 0.75 when the true mean change is 4 units. What is the Type II error probability under that same condition?
In words, what does power 0.60 mean for repeated studies when a specified alternative is true?
Why is “the probability the null hypothesis is true is 0.60” not a correct interpretation of power 0.60?
A study reports power under a true success probability of 0.65. Does that value automatically describe power when the true success probability is 0.55? Explain.
A test does not reject \(H_0\), even though it had high power under a particular alternative. Is this outcome impossible? Explain.