Type I Error as a Long-Run Rate

What a Type I Error Measures

A hypothesis test is a rule for deciding whether the observed data are sufficiently inconsistent with a null hypothesis to reject it. A Type I error occurs when that rule rejects the null even though the null model is true. Its probability describes how often the rule would make that error across repeated applications under a specified true null model. It is not, by itself, the probability that a particular rejection is wrong.

Definition: Let $R$ be the rejection region of a test, meaning the set of data outcomes for which the test rejects $H_0$. If a fixed parameter value $\theta_0$ satisfies $H_0$, the Type I error probability at that value is $$ P_{\theta_0}(\text{reject }H_0)=P_{\theta_0}(X\in R). $$ It is the probability, under the specified true null model, that the data fall in the rejection region.

The probability is always tied to a model and a decision rule. To calculate it, specify the null model for the data, identify exactly which outcomes lead to rejection, and find the probability of those outcomes under that model. A significance level such as 0.05 is not automatically the exact Type I error probability for every possible test: the rule and its null distribution determine the actual probability.

Why It Is a Long-Run Rate

Imagine repeating the same study independently many times, each time using the same test and sampling from the same fixed model in which $H_0$ is true. Let $I_j=1$ if replication $j$ rejects $H_0$, and let $I_j=0$ otherwise. If the Type I error probability in each replication is $\alpha$, then each $I_j$ is a Bernoulli random variable with success probability $\alpha$. The fraction of replications that reject is the average of these indicators.

\bar I_m=\frac{1}{m}\sum_{j=1}^{m} I_j, \qquad E[\bar I_m]=\alpha, \qquad \operatorname{Var}(\bar I_m)=\frac{\alpha(1-\alpha)}{m}.

The expectation of the fraction is $\alpha$, and its variance decreases as the number of independent replications $m$ increases. By the law of large numbers, the fraction of rejections approaches $\alpha$ as $m$ grows. This is the repeated-sampling meaning of the Type I error probability. It does not assert that exactly an $\alpha$ fraction of any finite collection of studies will reject.

Fix the null model.
Specify the parameter value or distribution assumed to be true for the calculation.

Fix the test rule.
State which data outcomes lead to rejection. This defines the rejection region.

Find the null probability of rejection.
Calculate the probability that data generated under the fixed null model fall in that rejection region.

Interpret the result across repetitions.
It is the long-run proportion of rejections when the same model and rule are used repeatedly, not a post-data probability that one decision is erroneous.

Worked Examples

Worked Example: An Exact Type I Error Rate for a Binomial Test

A hypothetical service center is studying the proportion $p$ of incoming calls that require escalation. In each study, five calls are selected independently, and $X$ is the number requiring escalation. The null hypothesis is $H_0:p=0.20$. The rule is to reject $H_0$ if $X\geq 3$. Find the Type I error probability and interpret it over 1,000 independent studies conducted under this null model.

Model and method. Under the fixed null model, the five calls are independent and each has escalation probability 0.20. Thus $X\sim\operatorname{Binomial}(5,0.20)$. A Type I error occurs precisely when $X$ is 3, 4, or 5, so add those three null probabilities.

\begin{aligned} \alpha &=P_{0.20}(X\geq 3)\\ &=P(X=3)+P(X=4)+P(X=5)\\ &=\binom{5}{3}(0.20)^3(0.80)^2 +\binom{5}{4}(0.20)^4(0.80) +\binom{5}{5}(0.20)^5\\ &=10(0.008)(0.64)+5(0.0016)(0.80)+0.00032\\ &=0.0512+0.0064+0.00032\\ &=0.05792. \end{aligned}

The Type I error probability is $0.05792$, or 0.0579 to four decimal places. In 1,000 independent studies conducted with this same null model and rule, the expected number of rejections is $1{,}000(0.05792)=57.92$, or about 58. The number in any particular set of 1,000 studies need not equal 58.

Check. The rejection region contains the three possible counts 3, 4, and 5, and their probabilities sum to less than 1. The calculation uses $P(X\geq 3)$, not just $P(X=3)$. The common mistake this avoids is treating a discrete rejection rule as though only its boundary outcome caused rejection.

Worked Example: A One-Sided Normal-Mean Test

A hypothetical analyst measures processing time in a stable operation. Each observation is independent and normally distributed with mean $\mu$ and known standard deviation 6 minutes. A study takes $n=9$ observations and rejects $H_0:\mu=50$ if the sample mean exceeds 53 minutes. Find the Type I error probability under $H_0$, then give the expected number of rejections in 1,000 independent studies under that null model.

Model and method. When $H_0$ is true, the observations are independent normal variables with mean 50 and standard deviation 6. Therefore the sample mean has mean 50 and standard error $6/\sqrt{9}=2$. Standardize the rejection threshold using this standard error.

\begin{aligned} \alpha &=P_{H_0}(\bar X>53)\\ &=P\left(\frac{\bar X-50}{2}>\frac{53-50}{2}\right)\\ &=P(Z>1.5)\\ &=1-\Phi(1.5)\\ &\approx 1-0.9332\\ &\approx 0.0668. \end{aligned}

The Type I error probability is approximately 0.0668, or 6.68%. In 1,000 independent studies under the fixed null model, the expected number of rejections is approximately $1{,}000(0.0668)=66.8$, or about 67.

Check. The threshold 53 is 3 minutes above the null mean, which is $3/2=1.5$ standard errors above it. Since rejection is in the upper tail, the relevant probability is $P(Z>1.5)$, not $P(Z<1.5)$. This test’s actual Type I error probability is about 0.0668; it should not be described as 0.05 merely because 0.05 is a commonly used significance level.

Worked Example: A Two-Sided Standard-Normal Rule

A hypothetical quality study uses a test statistic $Z$ that has the standard normal distribution when $H_0$ is true. The test rejects if $|Z|>2$. Calculate the Type I error probability and the expected number of rejections in 400 independent studies conducted under the same true null model.

Model and method. Under the fixed null model, $Z\sim N(0,1)$. The rejection region is the union of the two disjoint tails $Z<-2$ and $Z>2$. By symmetry, their probabilities are equal.

\begin{aligned} \alpha &=P(Z<-2)+P(Z>2)\\ &=2\Phi(-2)\\ &=2\bigl(1-\Phi(2)\bigr)\\ &\approx 0.0455003\\ &\approx 0.0455 \quad\text{to four decimal places.} \end{aligned}

Thus the Type I error probability is approximately 0.0455. The expected number of rejections in 400 independent studies is $400(0.0455003)\approx18.2001$, or about 18.2. A realized collection of 400 studies must have an integer number of rejections, so 18.2 is an expectation, not a possible exact count.

Check. The two tails are both included, and the unrounded tail probability is doubled before rounding. The common mistake is to round $P(Z>2)$ prematurely and then report a more precise doubled value than that rounded input supports.

What the Rate Does Not Mean

A Type I error probability is calculated conditional on a null model being true. For example, $\alpha=0.05$ means that if the null model is true and the same test is applied repeatedly under its assumptions, about 5% of those applications will reject in the long run. It does not mean that after one rejection there is a 5% probability that the null hypothesis is true, or that the particular rejection has a 5% probability of being wrong.

Once a particular data set has been observed and a nonrandomized test has been applied, the decision is determined: it either rejects or does not reject. The Type I error probability describes the chance of a rejection before observing the data, under the assumption that the null model is true. Inferring the probability that a rejected null is actually true would require information beyond the Type I error rate, including assumptions about the possible models and how plausible they are. The Type I error rate alone does not provide that probability.

Statement	Interpretation
“The Type I error probability is 0.05.”	Under the stated true null model and test rule, the rejection probability is 0.05.
“About 5% of repeated studies reject when this null model is true.”	A long-run interpretation of that same probability, assuming comparable independent repetitions.
“There is a 5% chance this particular rejection is wrong.”	Not implied by the Type I error probability alone.
“Exactly 5 of every 100 studies must reject.”	Not guaranteed; the rate describes a probability and a long-run tendency.

A further qualification is needed when the null hypothesis is composite, meaning it allows more than one parameter value or model. The rejection probability can then differ across values that satisfy $H_0$. A Type I error rate must be tied to a particular null value, or reported as a bound across the null values, rather than treated as automatically identical everywhere in the null hypothesis. The examples above used fixed null models so that the probability being calculated was unambiguous.

Key takeaway: A Type I error is a rejection under a true null model. Its probability is the null-model probability of the test’s rejection region and, under repeated comparable sampling, the long-run rejection rate. It is not the probability that a particular decision is wrong.

Check Your Understanding

For each problem, calculate or interpret the Type I error under the stated fixed null model. Give probabilities to four decimal places when needed.

A hypothetical audit samples four independent records, each having an error probability of 0.25 under $H_0$. It rejects if at least two sampled records have errors. Find the Type I error probability. (Answer: 0.2617)
Under a fixed null model, $X\sim\operatorname{Binomial}(6,0.10)$, and the rule rejects when $X\geq2$. Find the Type I error probability. (Answer: 0.1143)
Independent normal measurements have standard deviation 8. A test uses $n=16$, has $H_0:\mu=40$, and rejects when $\bar X<36$. Find the Type I error probability under $H_0$. (Answer: 0.0228)
A test statistic has a standard normal distribution under $H_0$ and rejects when $|Z|>2.5$. Find the Type I error probability and the expected number of rejections in 600 independent studies conducted under this true null model. (Answer: 0.0124; approximately 7.45 rejections)
A test has Type I error probability 0.01 under a specified true null model. What does this number say about repeated applications, and does it give the probability that a particular rejection is wrong? (Answer: About 1% of comparable repeated applications reject in the long run under that true null model; it does not give the probability that a particular rejection is wrong.)

What You'll Learn