From a Decision Rule to a Type I Error Probability
A hypothesis test is a rule for deciding whether the observed data are sufficiently inconsistent with a null hypothesis to reject it. Once the rule is fixed, its Type I error probability follows from the probability model under the null: it is the chance that data generated under a null parameter value fall in the rejection region. The word chance matters. A test’s error probability is calculated before the data are observed, under a specified model; it is not a probability that the null hypothesis is true.
The central question is how to make a rejection rule whose Type I error probability does not exceed a chosen bound. For a simple null hypothesis there is one null parameter value to check. For a composite null hypothesis, there are several possible null values, and the rule must be assessed over all of them.
The notation \(P_\theta\) means probability calculated using the distribution specified by parameter value \(\theta\). If a test is written using a statistic \(T(X)\), with a rule such as “reject when \(T(X)\ge c\),” then the data rejection region is \(R=\{x:T(x)\ge c\}\). The test statistic is a convenient way to describe the region; it does not change the definition of Type I error.
Why the Worst-Case Null Probability Is the Size
A composite null hypothesis contains multiple parameter values that could generate the data while \(H_0\) is true. A rejection rule may be more likely to reject at some of those values than at others. Checking just one convenient null value is not enough: the largest probability of a false rejection is the relevant bound.
The equivalence follows directly from the definition of a supremum. If the supremum is at most \(\alpha\), every probability included in that supremum is at most \(\alpha\). Conversely, if every null probability is at most \(\alpha\), their supremum is also at most \(\alpha\). This is why test size is a bound on Type I error under the entire null, rather than a calculation only at one selected null value.
For a discrete statistic, the probability of the rejection region is a sum of the probabilities of its outcomes. For a continuous statistic with density \(f_\theta\), it is the integral of that density over the region. These are two ways of calculating the same probability, not different definitions of size.
Record every parameter value allowed by \(H_0\). For a composite null, do not replace the set with its boundary until you have justified that the boundary is worst-case.
For example, “reject when \(T(X)\ge c\)” means \(R=\{x:T(x)\ge c\}\).
Find \(P_\theta(X\in R)\) using the model at a null parameter value \(\theta\).
Maximize over \(\theta\in H_0\), or justify a simpler way of identifying the supremum.
The test controls Type I error at level \(\alpha\) only if the largest null rejection probability is at most \(\alpha\).
Worked Applications
Worked Example: A Discrete Count Test Under a Composite Null
A hypothetical laboratory inspects five independent units from a production process. Each unit has probability \(p\) of requiring rework, and \(X\) is the number requiring rework. Thus \(X\sim\operatorname{Binomial}(5,p)\). The test is \(H_0:p\le 0.20\) against \(H_1:p>0.20\), with the rule “reject when \(X\ge3\).” Find the test size and determine whether it has level \(0.06\).
Method. Calculate \(P_p(X\ge3)\) and find its supremum over the null values \(0\le p\le0.20\). For a binomial count with a fixed sample size, the probability of reaching at least three successes increases with \(p\). One justification is to take independent uniform random variables \(U_1,\ldots,U_5\) and define each success indicator as \(I(U_j\le p)\). Increasing \(p\) cannot decrease any indicator or their sum. Therefore the largest rejection probability over the null occurs at \(p=0.20\).
At the boundary, \(X\sim\operatorname{Binomial}(5,0.20)\). The rejection probability is the sum of the probabilities of \(X=3,4,5\):
The size is \(0.05792\), or \(0.0579\) to four decimals. Since \(0.05792\le0.06\), this rule has level \(0.06\). It is slightly conservative: its size is below the permitted level, not equal to it.
Common mistake avoided: Checking the null probability at an arbitrary value such as \(p=0.10\) would not establish control over all \(p\le0.20\). The size uses the largest probability over the composite null, which here occurs at the boundary.
Worked Example: A One-Sided Normal Test With a Strictly Smaller Size Than Its Level
Suppose a hypothetical monitoring system produces independent measurements \(X_1,\ldots,X_{16}\), each with distribution \(N(\mu,4)\). The variance is \(4\), so the standard deviation of each measurement is \(2\). Test \(H_0:\mu\le0\) against \(H_1:\mu>0\) by rejecting when \(\bar X>1\). Find the size and decide whether the rule has level \(0.05\).
Method. Under any value of \(\mu\), the sample mean has distribution \(\bar X\sim N(\mu,4/16)=N(\mu,0.25)\), so its standard deviation is \(0.5\). For a null value \(\mu\le0\), standardization gives
As \(\mu\) decreases below zero, the standardized cutoff \((1-\mu)/0.5\) increases, and the upper-tail probability decreases. The supremum over \(\mu\le0\) is therefore at \(\mu=0\). At that boundary,
Thus the test size is \(0.0228\) to four decimals. Because \(0.0228\le0.05\), it has level \(0.05\), but it does not use the entire allowance. The cutoff of \(1\) makes rejection less likely under the null than a cutoff calibrated to a \(0.05\) boundary probability.
Common mistake avoided: Calling the size \(0.05\) merely because \(0.05\) is the target level confuses an upper bound with the actual worst-case rejection probability. The actual size is \(0.0228\).
Worked Example: Counting Both Tails in a Two-Sided Test
A hypothetical instrument produces independent readings \(X_1,\ldots,X_9\), each with distribution \(N(\mu,9)\). Test the simple null \(H_0:\mu=0\) using the rule “reject when \(\bar X\le-2\) or \(\bar X\ge2\).” Find the size and determine whether the test has level \(0.05\).
Method. The sample mean has distribution \(\bar X\sim N(\mu,9/9)=N(\mu,1)\). Under the simple null, \(\bar X\sim N(0,1)\). The rejection region includes two disjoint tails, so the total rejection probability is the sum of the two tail probabilities:
The null is simple, so this one probability is the size: \(0.0455\) to four decimals. Since \(0.0455\le0.05\), the test has level \(0.05\).
Common mistake avoided: Reporting \(0.0228\) as the size counts only the upper tail. A two-sided rejection region includes both tails, and both contribute to the probability of a Type I error.
What Size Does—and Does Not—Say
A level-\(\alpha\) rule guarantees that, for every parameter value in the null hypothesis, the probability of rejection is no greater than \(\alpha\). The guarantee concerns the rejection region and the assumed probability model. It does not say that a particular observed rejection has probability \(\alpha\), and it does not assign probability \(\alpha\) to the null hypothesis.
The distinction between level and size is especially important when a test is discrete. The rejection region may jump from one attainable probability to another, so a test can have size below the requested level. In contrast, for some continuous models a cutoff can be chosen to make the boundary rejection probability equal to the target. In either case, the same criterion applies: calculate the probability of entering the rejection region under the null, then check the largest such probability over all null parameter values.
Common Exam Traps
- Using only one null value for a composite null. A calculation at the boundary is valid only after establishing that it gives the largest null rejection probability, or otherwise showing that it equals the supremum.
- Equating the level with the size. Level \(\alpha\) means size at most \(\alpha\). The test may be conservative and have a smaller size.
- Forgetting part of the rejection region. Add the probabilities of all disjoint rejection outcomes, including both tails in a two-sided test.
- Using the alternative distribution to calculate Type I error. Type I error is evaluated at parameter values in \(H_0\), not at an alternative value.
- Ignoring the model assumptions. For example, the binomial calculation requires independent trials with the same success probability; the normal sample-mean calculations above require independent normal observations with the stated variance.
Check Your Understanding
For each test, identify or calculate its size and compare it with the stated level. Answers are rounded to four decimals where needed.
- A hypothetical screening process has \(X\sim\operatorname{Binomial}(4,p)\). Test \(H_0:p\le0.10\) by rejecting when \(X\ge2\). Find the size and decide whether the rule has level \(0.06\). (Answer: size \(0.0523\); yes, level \(0.06\).)
- Independent measurements have distribution \(N(\mu,16)\), and there are \(4\) measurements. For \(H_0:\mu\le3\), reject when \(\bar X\ge5\). Find the size and state whether this is a level \(0.16\) test. (Answer: size \(P(Z\ge1)=0.1587\); yes, level \(0.16\).)
- A hypothetical monthly count has distribution \(Y\sim\operatorname{Poisson}(\lambda)\). Test \(H_0:\lambda\le1.5\) by rejecting when \(Y\ge4\). Use the monotonicity of the Poisson upper tail to identify the worst-case null value and calculate the size. (Answer: the worst case is \(\lambda=1.5\); size \(1-e^{-1.5}(1+1.5+1.125+0.5625)\approx0.0656\).)
- Independent observations have distribution \(N(\mu,25)\), with sample size \(25\). Under the simple null \(H_0:\mu=10\), reject when \(|\bar X-10|\ge2.5\). Find the size and decide whether the test has level \(0.02\). (Answer: size \(2[1-\Phi(2.5)]\approx0.0124\); yes, level \(0.02\).)
- A hypothetical count \(X\) has distribution \(\operatorname{Binomial}(6,p)\). For \(H_0:p\le0.30\), reject when \(X\ge4\). Explain why the boundary is worst-case and calculate the size. (Answer: the upper-tail probability increases with \(p\), so the worst case is \(p=0.30\); size \(P_{0.30}(X\ge4)=0.059535+0.010206+0.000729=0.0705\).)