The Union Bound for Repeated False Positives

Why Bound the Chance of Any False Rejection?

When a trial is examined at several scheduled looks, each test can reject the trial-wise null hypothesis \(H_0\). If \(H_0\) is true, a rejection at any look is a false rejection. The trial-wide error event is therefore not limited to a false rejection at the final look: it is the event that at least one of the looks rejects.

The probability of this event depends on how the look-specific rejection events relate to one another. The union bound gives a useful guarantee when their joint behavior is unknown: the chance of at least one false rejection is no greater than the sum of the individual false-rejection probabilities. It does not require independent tests.

Definition: For a fixed monitoring plan with \(K\) looks, let \(A_k\) be the event that the test at look \(k\) rejects \(H_0\), for \(k=1,\ldots,K\). Under the trial-wise null, the event of any false rejection is \(A_1\cup A_2\cup\cdots\cup A_K\). If \(P(A_k)\leq\alpha_k\) for every look, where \(\alpha_k\) is that look’s rejection-probability bound under \(H_0\), then the union bound states that \(P(A_1\cup\cdots\cup A_K)\leq\sum_{k=1}^K\alpha_k\).

Here, each probability is evaluated under the null model for the trial. A per-look bound means the test at that look has probability at most \(\alpha_k\) of rejecting when \(H_0\) is true. If the null is composite, the requirement is that this bound hold for every parameter value in \(H_0\); controlling only at one convenient null value is not enough to guarantee control over the whole null hypothesis.

Why the Union Bound Holds

The key fact is that whenever at least one event \(A_k\) occurs, the number of events that occur is at least one. Write \(I(A_k)\) for an indicator: it equals 1 when \(A_k\) occurs and 0 otherwise. For every possible outcome,

I(A_1\cup\cdots\cup A_K)\leq\sum_{k=1}^{K}I(A_k).

If no event occurs, both sides are zero. If one or more events occur, the left side is one and the right side is at least one. Taking expectations gives the probability inequality, because the expected value of an event indicator is the probability of that event and expectation is additive:

P(A_1\cup\cdots\cup A_K) =E[I(A_1\cup\cdots\cup A_K)] \leq E\left[\sum_{k=1}^{K}I(A_k)\right] =\sum_{k=1}^{K}P(A_k) \leq\sum_{k=1}^{K}\alpha_k.

This proof uses no assumption about independence, positive association, or negative association. Overlap between events can make their union probability smaller than their summed probabilities, but it cannot make the union probability larger than the sum. The union bound is an upper bound, not usually an equation.

Worked Example: Four Looks With Equal Per-Look Bounds

A hypothetical trial has four scheduled looks. At each look, the test has false-rejection probability at most 0.025 under the trial-wise null. Find a guaranteed upper bound for the probability of any false rejection.

Let \(A_k\) be the false-rejection event at look \(k\). The four individual bounds are \(\alpha_1=\alpha_2=\alpha_3=\alpha_4=0.025\). Applying the union bound,

P(A_1\cup A_2\cup A_3\cup A_4) \leq 0.025+0.025+0.025+0.025 =4(0.025)=0.100.

Thus the probability of at least one false rejection is no greater than 0.100, or 10.0%, under the stated per-look bounds. This is a guarantee, not a claim that the probability equals 10.0%. The bound applies whether or not the four tests are independent.

Using Per-Look Error Budgets

The inequality is especially useful when a design assigns different rejection probabilities to different looks. If the sum of the assigned bounds is no greater than a desired trial-wide limit \(\alpha\), then the probability of any false rejection is also no greater than \(\alpha\). This is the basic logic of Bonferroni error control applied across looks.

\sum_{k=1}^{K}\alpha_k\leq\alpha \quad\Longrightarrow\quad P(A_1\cup\cdots\cup A_K)\leq\alpha.

The allocation need not be equal. A designer might assign smaller per-look bounds to some looks and larger bounds to others, as long as each test meets its assigned bound and the total does not exceed the target. This inequality alone does not determine which allocation is best for a particular trial; it provides a transparent way to ensure the overall false-rejection probability stays within the stated limit.

Worked Example: Unequal Look-Specific Error Budgets

A hypothetical monitoring plan has three looks and a trial-wide error target of 0.050. Suppose the per-look rejection probabilities under \(H_0\) are bounded by 0.010 at the first look, 0.015 at the second, and 0.020 at the third. Check whether these bounds guarantee the target.

Add the three bounds and compare the total with 0.050:

0.010+0.015+0.020=0.045\leq0.050.

Therefore,

P(A_1\cup A_2\cup A_3)\leq0.045.

The plan meets the 0.050 target by this argument, with 0.005 of the target left unused by the bound. The conclusion depends on each stated per-look probability bound actually holding under the null; merely writing down the three numbers as intended thresholds does not establish that the tests have those rejection probabilities.

The bound can also be useful when per-look probabilities are known or estimated from the specified null model rather than set directly as nominal significance levels. The same arithmetic applies, but the probability statements must correspond to the tests and monitoring plan actually used.

Worked Example: Bounding Error From Look-Specific Probabilities

For a hypothetical three-look procedure, suppose a valid calculation under \(H_0\) gives marginal false-rejection probabilities of 0.008, 0.012, and 0.009 at the three looks. Find the union-bound guarantee and explain what it does and does not establish.

Add the probabilities:

P(A_1\cup A_2\cup A_3) \leq 0.008+0.012+0.009 =0.029.

The chance of at least one false rejection is at most 0.029, or 2.9%, under the assumed null model. This does not establish that the chance is exactly 2.9%. Since the union contains each individual event, it must have probability at least as large as the largest individual probability, which here is 0.012. With only these marginal probabilities, the union probability is therefore bounded below by 0.012 and above by 0.029; its exact value is not determined.

Assumptions and Practical Checks

The union bound itself is mathematically general. Its usefulness for a trial depends on correctly defining the events and verifying their probability bounds. For the result above to give a trial-wide guarantee:

The monitoring plan must identify the looks being included, so that the collection \(A_1,\ldots,A_K\) covers every rejection opportunity to which the claim refers.
Every \(A_k\) must represent rejection of the same true trial-wise null, or the probability statement must otherwise specify the null configuration being controlled.
Each claimed \(\alpha_k\) must genuinely bound \(P(A_k)\) under that null. It is not enough that \(\alpha_k\) is called a significance level if the test is not calibrated to have the stated null rejection probability.
The probability bounds must apply to the actual procedure, including the analysis rule and any relevant choices made before or during monitoring.

For a fixed schedule, one diagnostic is to list every planned look, its rejection rule, and its null rejection-probability bound, then check that the sum is within the desired limit. A missing look or an unaccounted rejection rule can invalidate the claimed total bound. If the schedule or test rule is changed, the list and its probability guarantees need to be checked again.

A particular caution concerns data-dependent decisions. The union inequality still holds for a set of events, but a nominal per-look threshold does not automatically guarantee the claimed marginal probability when the look or test is selected using the accumulated data. The relevant \(P(A_k)\) must be justified for the actual procedure, not simply borrowed from a fixed-look test calculation.

What the Bound Says—and What It Does Not

The union bound is a conservative, dependence-robust way to control any-look error. It is attractive when a simple guarantee is needed and no joint model for the tests is being used. Its cost is that the sum can be substantially larger than the actual probability of the union, particularly when rejection events overlap. If the sum exceeds the desired trial-wide target, this does not prove the design exceeds that target; it means this particular bound does not certify the target.

An exact calculation requires information about the joint behavior of the tests. Under the independent-test assumption, a product calculation can give an exact any-look probability from the individual probabilities. That independence-based calculation is a different approach and is appropriate only when its assumption is justified. In contrast, the union bound needs only valid marginal bounds, so it remains applicable when the joint behavior is unknown. It may be less precise, but it does not acquire an independence assumption.

The union bound also has a limitation in what it controls: it bounds the probability of at least one false rejection over the specified events. It does not describe when the first rejection occurs, how often a particular look is responsible, or the probability of rejection when the null is false. Those are different questions.

Key takeaway: If each of \(K\) scheduled tests has null rejection probability at most \(\alpha_k\), then the probability of any false rejection is at most \(\sum_{k=1}^{K}\alpha_k\), regardless of dependence. A total error target \(\alpha\) is guaranteed whenever that sum is no greater than \(\alpha\).

Common Mistakes

Adding probabilities and calling the result exact. The union bound gives an upper limit, not generally the union probability. State “at most” unless an exact joint calculation has been made.
Assuming independence is needed. Independence is not required for the inequality. It may be needed for a separate exact product calculation, but not for the union bound.
Using per-look labels without checking calibration. A nominal threshold is useful only if the probability of rejection under the null is actually bounded as claimed for the test being used.
Counting too few opportunities. The bound covers the events included in the union. If an additional analysis can also trigger rejection, it must be accounted for in the error-control argument.
Treating a failed bound as proof of excessive error. A sum above the target makes the bound inconclusive for that target; the true union probability may still be smaller.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Five looks each have false-rejection probability at most 0.008 under \(H_0\). What upper bound does the union bound give for any false rejection?
Three look-specific bounds are 0.010, 0.020, and 0.025. Does the union bound certify a trial-wide target of 0.050?
True or false: the union bound can be used only if the look-specific tests are independent. Explain.
Two events have probabilities 0.03 and 0.04. What upper bound does the union bound give for their union, and why is that not necessarily the exact probability?

Solutions. (1) Add the five bounds: \(5(0.008)=0.040\). The probability of any false rejection is at most 0.040, or 4.0%. (2) The sum is \(0.010+0.020+0.025=0.055\), which exceeds 0.050. This bound does not certify the target, although it does not prove that the true union probability exceeds 0.050. (3) False. The indicator inequality and expectation argument require no independence assumption. (4) The bound is \(0.03+0.04=0.07\). It is not necessarily exact because the events may overlap; the union counts an outcome only once even when both events occur.