Common Misconceptions About Alpha, Beta, and Power

Probability Depends on What We Assume Is True

The words “probability of rejecting” can sound straightforward, but a test’s rejection probability depends on the actual population condition. In “Statistical Conclusions and Possible Mistakes in Context,” we separated a test decision from the truth about the population. Here, we sharpen that distinction: alpha, beta, and power are conditional probabilities. Each describes the chance of a test decision given a specified truth.

For a one-proportion test with null hypothesis \(H_0:p=p_0\), alpha describes the probability of rejecting \(H_0\) when \(p=p_0\) is true. For a specified alternative value \(p_1\), beta describes the probability of failing to reject \(H_0\) when \(p=p_1\) is true. Power is the probability of rejecting \(H_0\) when that specified alternative value is true.

The vertical bar in probability notation means “given that.” Thus, alpha is a probability about the test’s decision given that the null model is true. It is not a probability about whether the null hypothesis is true. Likewise, power is a probability about the test’s decision given a particular alternative value, not a general probability that the research claim is correct.

For a particular test result, the population proportion is unknown. We therefore do not use alpha to say how likely it is that \(H_0\) is true after seeing the data. A significance test does not calculate that probability. Its p-value answers a different question: assuming \(H_0\) is true, how likely is a result at least as extreme as the one observed? As explained in “Significance Level as the Probability of a Type I Error,” alpha is chosen as a standard for the testing procedure; the p-value is calculated from the sample.

Alpha Is Not the Probability That the Null Is True

A common misconception is, “If \(\alpha=0.05\), there is a 5% chance that the null hypothesis is true.” That reverses the condition. Alpha does not describe the probability of a hypothesis being true. It describes the probability that the test rejects when the null hypothesis is true.

Think about repeating the same testing procedure many times under the same conditions, with the null hypothesis actually true each time. If the procedure has significance level \(\alpha=0.05\), its long-run probability of rejection is 0.05. In many repetitions, about 5% of the tests would reject a true null, although the exact percentage in a finite set of repetitions need not be exactly 5%.

That long-run statement does not tell us whether the null hypothesis is true in any one investigation. Nor does it mean that every rejection has a 5% probability of being wrong. Whether a particular rejection is a Type I error depends on the actual population truth, which the test generally does not reveal.

Power Is Conditional on a Specified Alternative

A second common misconception is, “Power is the probability of rejecting.” That wording is incomplete. A test can have different probabilities of rejection at different true population proportions. To state power, specify the alternative value being treated as true.

Suppose a test assesses \(H_0:p=p_0\), and we want to describe its ability to detect a particular change. We select a value \(p_1\) that makes the null hypothesis false. The probability of failing to reject \(H_0\) when \(p=p_1\) is \(\beta(p_1)\). The probability of rejecting under that same condition is the power at \(p_1\):

The phrase “at \(p_1\)” matters. A test designed to detect a large difference from \(p_0\) may have high power for that value but lower power for a value closer to \(p_0\). As discussed in “How Effect Size Affects Power,” power generally increases when the true alternative is farther from the null in the direction of the alternative, with the sample size and significance level held fixed.

Power is not the probability that the alternative hypothesis is true. It is also not a guarantee that the test will reject in one particular sample. It describes a long-run probability under the stated alternative condition.

“Probability of Rejecting” Needs a Truth Condition

The test procedure has a rejection rule, but the probability that a sample lands in the rejection region changes with the true population value. Under the null value, that probability is alpha. Under a specified false-null value, it is power at that value. If the true value is not specified, “the probability of rejecting” does not identify one particular probability.

The following hypothetical operating characteristics make the distinction visible. Suppose a test of \(H_0:p=0.40\) against \(H_a:p>0.40\) has significance level 0.05, and its calculated power is 0.30 when \(p=0.45\) and 0.80 when \(p=0.55\). These values describe one illustrative procedure; they are not results from an actual study.

All three entries concern the chance of rejection, but they condition on different true proportions. The first is alpha; the other two are powers. The larger power at \(p=0.55\) reflects that this value is farther above the null value than \(p=0.45\), making it easier for the test to detect the increase.

Do Not Confuse Alpha, Power, and the P-Value

These three quantities all involve probabilities, but they answer different questions. Keeping their conditions and roles separate prevents many interpretation errors.

• Alpha: Before collecting data, the investigator chooses a significance level. It is the probability that the testing procedure rejects \(H_0\) when \(H_0\) is true.
• Power: For a specified true value that makes \(H_0\) false, power is the probability that the procedure rejects \(H_0\).
• P-value: After observing sample data, the p-value is calculated under the assumption that \(H_0\) is true. It measures how unusual the observed result, or a more extreme result, would be under that null model.

Alpha is a chosen decision standard; the p-value is evidence from the observed sample compared with that standard. Power describes how often the procedure would reject under a specified alternative. A p-value is not the probability that \(H_0\) is true, and alpha is not the p-value for every sample. Power is not the probability that a rejection is correct.

The distinction also clarifies why “fail to reject” is not proof of the null. A test can fail to reject when the null is true, which is a correct decision, or when a false null is true, which is a Type II error. How likely the latter is depends on beta at the actual alternative value.

Common Mistakes and AP Exam Tip

• Reversing the condition in alpha: “Alpha is the probability that \(H_0\) is true” is incorrect. Full-credit wording says alpha is the probability of rejecting a true \(H_0\), or \(P(\text{reject }H_0\mid H_0\text{ is true})\).
• Leaving the alternative value out of a power interpretation: “The test has 80% power” should be tied to a specified true value, such as “when the population proportion is 0.70.” Power can differ at other false-null values.
• Calling power the probability of any rejection: State what is assumed true. Under the null, the rejection probability is alpha; under a particular alternative, it is power at that alternative.
• Calling power the probability that the alternative is true: Power conditions on the alternative being true. It does not estimate the chance that this condition holds in the population.
• Mixing up power and beta: At the same specified alternative, \(\beta\) is the probability of failing to reject, while power is the probability of rejecting. They add to 1.
• Confusing alpha with a p-value: Alpha is set as a threshold before the sample result is evaluated. The p-value comes from the observed data under the null model. Compare them to make a test decision; do not describe them as the same quantity.
• Using “the chance this decision is wrong”: Neither alpha nor power directly gives that probability for an individual test result. They describe conditional long-run behavior under specified truths.

On an AP response, include the condition in your sentence: “If the true proportion is 0.70, the test has a 0.80 probability of rejecting \(H_0\).” For alpha, name the null truth: “If \(H_0\) is true, the test has a 0.05 probability of rejecting it.” Conditional wording makes clear that these probabilities describe the procedure, not certainty about the unknown population.