Significance Level as the Probability of a Type I Error

From Type I Error to Significance Level

In “Two Possible Errors in a Significance Test” and the tutorials that followed, you learned that a Type I error is rejecting a true null hypothesis. The significance level, written \(\alpha\), connects that definition to the chance of making this error. It describes how often a testing procedure is designed to reject \(H_0\) when \(H_0\) is true.

The vertical bar in \(P(\text{reject }H_0\mid H_0\text{ is true})\) means “given that.” So this notation asks: if the null hypothesis is in fact true, what is the probability that the test procedure will reject it? The answer is the probability of a Type I error.

A significance test uses sample data to decide whether to reject \(H_0\) or fail to reject it. Before looking at the data, the researcher chooses \(\alpha\), such as \(0.05\). The test is then set up so that, when the null model is true, the chance of getting a result that leads to rejection is controlled at the chosen level.

This is a long-run description of the procedure, not a prediction that a particular test will make a fraction of an error. For any one completed test, the decision either is or is not a Type I error; we usually do not know the actual truth with certainty. Alpha tells us how often the procedure would make that error over repeated uses when the null hypothesis is true.

What Does Alpha = 0.05 Mean?

Suppose a study tests whether a new community program increases the population proportion of residents who sort food scraps for composting. Let \(p\) be that population proportion after the program is introduced. The hypotheses are \(H_0:p=0.30\) and \(H_a:p>0.30\), and the test uses \(\alpha=0.05\).

If \(H_0\) is true, the population proportion is \(0.30\). In repeated applications of this same testing procedure under that null condition, the probability of rejecting \(H_0\) is \(0.05\). Those rejections would be Type I errors: the procedure would indicate convincing evidence that the proportion exceeds \(0.30\), although it does not.

“About 5%” is a long-run rate, not a promise about a particular batch of tests. If the procedure is repeated 100 times while the null is true, it does not have to produce exactly five Type I errors. A particular set of repetitions could produce fewer or more.

For a null hypothesis that covers a range of values, such as \(H_0:p\le 0.30\), the interpretation needs a small qualification. The test is designed to limit the probability of a Type I error to no more than \(0.05\) for values in the null range. The probability is often exactly \(0.05\) at the boundary, \(p=0.30\), and smaller for some values farther inside the null range. At the AP Statistics level, describe alpha as the procedure’s Type I error rate under the null model, and identify the relevant null value when the setting makes it important.

Worked Examples: Interpreting Alpha in Context

Alpha Is Not the Probability a Conclusion Is Wrong

The conditional wording matters. Alpha is \(P(\text{reject }H_0\mid H_0\text{ is true})\): it starts by assuming the null is true and asks how often the procedure rejects. It is not \(P(H_0\text{ is true}\mid\text{the test rejected})\). These probabilities condition on different information and are not interchangeable.

After a test rejects \(H_0\), alpha alone does not tell us the probability that this particular rejection is a Type I error. To make that probability claim, we would need information beyond the significance level, including how plausible the null was before the data and how well the procedure detects alternatives. AP test conclusions instead describe the evidence: for example, whether the sample provides convincing evidence for the alternative in context.

Alpha is also different from the p-value. Alpha is selected as a threshold before the test decision; the p-value is calculated from the observed data. The decision rule compares them: reject \(H_0\) when the p-value is less than or equal to \(\alpha\), and fail to reject \(H_0\) when it is greater than \(\alpha\). As the example shows, the same p-value can lead to different decisions under different preselected significance levels.

How to State Alpha Clearly

A strong interpretation names the null condition, the test decision that would be an error, and the population outcome being studied. Use conditional language: “If the true population proportion is …, the probability this procedure rejects \(H_0\) is …” Avoid describing alpha as a percentage of individuals, a probability that a hypothesis is true, or a guaranteed number of mistakes.

For the registration example, a complete statement is: “If the true proportion of families completing registration on time is \(0.70\), this procedure has a \(5\%\) probability of rejecting the null and concluding that the reminder increases the proportion.” This identifies the condition, the error, the probability, and the context.

Common Mistakes and AP Exam Tip

• Reversing the conditional probability: Alpha is the probability of rejecting \(H_0\) given that \(H_0\) is true. It is not the probability that \(H_0\) is true given that it was rejected.
• Calling alpha the probability that a specific conclusion is wrong: Alpha describes repeated use of a procedure under the null condition. It does not tell us whether one observed rejection is an error.
• Confusing alpha with the p-value: Alpha is selected as the significance threshold; the p-value comes from the sample. State which number is which before comparing them.
• Claiming an exact error count: An alpha of \(0.05\) does not guarantee exactly five Type I errors in every 100 tests. Say “about 5% in the long run” when the null is true.
• Leaving out the context: “There is a 5% chance of error” is vague. Full-credit wording identifies the false conclusion and the null condition, such as “If the true proportion is \(0.70\), there is a 5% chance the test will conclude it is greater than \(0.70\).”
• Overstating alpha for a composite null: When \(H_0\) covers a range, the procedure controls the Type I error probability at or below alpha across that range; it need not equal alpha at every null value.