Why the Significance Level Needs a Time Reference
A significance level is meaningful only after specifying which testing decision it describes. In a one-look test, \(\alpha\) is the prespecified Type I error rate for that test: under \(H_0\), its probability of rejecting \(H_0\) is at most \(\alpha\). In a monitored trial, investigators may test at several scheduled looks. The same nominal level can be applied at each look, but that does not by itself state the probability of at least one false rejection during the whole trial.
The distinction matters because a trial-wide claim is usually based on whether any scheduled test rejects. If \(H_0\) is true, even one such rejection is a false positive. The chance of that event depends on how the rejection events at the different looks relate to one another, not just on the nominal level written beside each test.
A procedure calibrated at nominal level \(\alpha\) at each look has \(P(A_k\mid H_0)\leq\alpha\) for each individual look, under the assumptions that make that one-look test valid. This is a statement about each event separately. It is not, without more information, a statement that the probability of the union of those events is \(\alpha\).
Two Looks: The Overlap Matters
For two looks, write \(A_1\) and \(A_2\) for their rejection events. The overall error event occurs if look 1 rejects, look 2 rejects, or both do. Adding the two per-look probabilities counts trials in which both looks reject twice, so the overlap must be subtracted once:
This identity is inclusion-exclusion for two events. It shows directly why per-look probabilities are not enough: calculating the overall probability also requires the probability of rejection at both looks. With more looks, the joint pattern of rejections matters as well.
Worked Example: Two Nominal Levels and a Known Overlap
Consider a hypothetical two-look monitoring schedule under \(H_0\). Suppose each look has a per-look Type I error probability of \(0.04\), and the probability that both looks reject is \(0.015\). Find the overall Type I error probability.
Let \(A_1\) and \(A_2\) denote rejection at the first and second looks. Apply the two-event identity, using the supplied overlap:
The overall false-positive probability is \(0.065\), or \(6.5\%\), for this hypothetical schedule and null. Each individual test has a \(4\%\) Type I error probability, but the chance of at least one false rejection is \(6.5\%\). The result uses the given overlap; it cannot be obtained from the two per-look probabilities alone.
For any schedule, the overall probability must be at least as large as the largest per-look probability, because a rejection at any particular look is one way for the “any rejection” event to occur. It can be larger, but it need not equal the sum of the per-look probabilities. The precise value depends on how much the rejection events overlap.
A Short Monitoring Schedule in Practice
A useful way to keep the concepts separate is to write down two different quantities. The first is the nominal level assigned to each scheduled test. The second is the trial-wide probability of one or more false rejections under \(H_0\). A schedule might specify three looks and use a nominal level of \(0.05\) at each. That specification gives the intended one-look calibration; by itself, it does not give an exact overall error rate.
Worked Example: Reading Rejection Counts Across Three Looks
Suppose a hypothetical exercise records the rejection patterns from 1,000 null trial trajectories for a three-look schedule. The patterns are summarized as follows. Each trajectory contributes to one row only.
| Rejection pattern | Number of trajectories |
|---|---|
| Reject at look 1 only | 15 |
| Reject at look 2 only | 15 |
| Reject at look 3 only | 10 |
| Reject at looks 1 and 2, but not 3 | 25 |
| No rejection at any look | 935 |
First check that the pattern counts cover all trajectories: \(15+15+10+25+935=1{,}000\). At look 1, there are \(15+25=40\) rejections, so the observed per-look rate is \(40/1{,}000=0.04\), or \(4\%\). Look 2 also has \(15+25=40\) rejections, again \(4\%\). Look 3 has 10 rejections, giving \(10/1{,}000=0.01\), or \(1\%\).
The number of trajectories with any rejection is \(15+15+10+25=65\). Thus the observed overall rate is:
The per-look percentages sum to \(4\%+4\%+1\%=9\%\), but that sum is not the overall rate: the 25 trajectories rejecting at both looks 1 and 2 are counted at both looks in that sum. The table makes the overlap explicit and gives an overall observed rate of \(6.5\%\). These are hypothetical frequencies, not an assertion that the true probability for any particular monitoring plan is exactly \(6.5\%\).
This example also illustrates a practical distinction. A rate calculated from a finite set of hypothetical or simulated trajectories is an observed proportion. The corresponding probability under \(H_0\) is a feature of the testing procedure’s null distribution. The observed proportion may help describe a procedure, but it is not automatically the exact probability.
What a Protocol’s Per-Look Level Does and Does Not Say
Suppose a protocol states that each of three tests will use a two-sided nominal significance level of \(0.025\). This tells the reader the one-look error criterion for each test, assuming the test is valid at that look and the stated null hypothesis is being considered. It does not specify the exact probability of rejecting at least once across all three tests. To calculate that overall probability, one needs adequate information about the joint rejection events, or an analysis procedure that directly controls the trial-wide error rate.
Worked Example: An Overall Rate Cannot Be Read Off the Protocol
A hypothetical protocol specifies looks at three prescheduled times and says that each look uses a test with nominal level \(0.025\). No joint rejection probabilities or trial-wide calibration are given. What overall Type I error probability can be reported from this information alone?
For each look \(k\), the nominal specification aims for \(P(A_k\mid H_0)\leq0.025\), if that look’s test assumptions hold. The protocol does not give \(P(A_i\cap A_j\mid H_0)\), nor does it otherwise state how the joint rejection pattern is controlled. Therefore an exact value for \(P(A_1\cup A_2\cup A_3\mid H_0)\) cannot be calculated from the nominal levels alone.
The appropriate conclusion is not “the overall level is \(0.025\)” and not “the overall level is \(0.075\).” The exact overall rate is unspecified by the information provided. A complete protocol needs to describe or justify how trial-wide error is handled; later material develops ways to formulate and assess that problem.
For comparison, a single prespecified test has one rejection event, so its one-look error criterion is also the error probability for that single decision. Repeating tests at nominal levels without a trial-wide adjustment retains the per-look interpretation but does not, by itself, retain the same interpretation for the full monitoring decision. Procedures designed to control overall error instead calibrate the sequence of decisions to a stated trial-wide target. The important point here is to identify which claim is actually supported, not to assume that two different claims share the same \(\alpha\).
Assumptions, Diagnostics, and Limitations
The definitions apply under the trial-wise null hypothesis: the null hypothesis being monitored is true throughout the trial. For the per-look interpretation, the test at each look must be valid for its stated data and analysis plan; in particular, its nominal calibration cannot be assumed if the test or timing was selected in an unaccounted-for way. The scheduled looks and the rejection rule also need to be clearly specified so that “any rejection” is a well-defined event.
When reviewing a plan or report, check whether it labels its rate as per-look or overall. Ask how many tests contribute to the decision, whether the stated level applies at each test or to the whole schedule, and what information supports the trial-wide claim. If only marginal levels are reported, the exact overall probability generally remains unknown. When rejection counts or probabilities are supplied, verify that the overall count includes each trajectory only once, even if it rejects at several looks.
This distinction has limits as a complete design analysis. Knowing that a schedule has several looks and a nominal level at each does not determine power, expected stopping time, or the exact overall error rate. Those require additional details about the test statistics, data, schedule, and decision rules. For this tutorial’s purpose, the essential diagnostic is narrower: do not interpret a one-look calibration as a trial-wide guarantee unless trial-wide control has been established.
Common Mistake
- Wrong reasoning: “There are three looks at \(0.05\), so the overall error is \(0.15\).” This adds marginal probabilities as if each false rejection occurred in a different set of trials. Trials that reject at multiple looks are counted more than once in that sum.
- Correction: The overall event is the union of the look-specific rejection events. Its probability depends on their joint pattern. Use supplied joint information or a procedure with stated overall-error control; do not infer an exact overall rate by adding nominal levels.
- Another mistake: “Each look is at \(0.05\), so the whole trial is at \(0.05\).” A one-look level does not establish that trial-wide claim either. State the scope of the level precisely.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a two-look plan, each test has a nominal level of \(0.03\). Does this information alone determine the overall Type I error probability?
- For two looks, suppose \(P(A_1\mid H_0)=0.05\), \(P(A_2\mid H_0)=0.04\), and \(P(A_1\cap A_2\mid H_0)=0.02\). Find the overall probability.
- A report gives an overall probability of \(0.07\) and a per-look probability of \(0.05\) at one look. Is that automatically contradictory?
- Why should the total number of rejections across looks not necessarily be used as the number of trajectories with any rejection?
Solutions. (1) No. The joint rejection pattern is not specified, so the exact probability of at least one rejection cannot be determined. (2) By inclusion-exclusion, \(0.05+0.04-0.02=0.07\), so the overall probability is \(0.07\). (3) No. The overall event can include rejections at other looks as well; an overall probability larger than the probability at one particular look is possible. (4) A trajectory can reject at more than one look. Counting each rejection separately would count that trajectory multiple times, whereas the overall event counts it once if there is at least one rejection.