Calculating Independent-Look Error Inflation

From Per-Look Error to Overall Error

When a trial is tested repeatedly, a false rejection at any one look can lead to a false-positive conclusion for the trial. The per-look significance level describes the error probability for one test under its null hypothesis. The overall Type I error describes the probability of at least one false rejection across the full set of looks when the trial-wide null hypothesis is true.

For independent looks, the overall probability can be calculated from the probability of avoiding rejection at every look. This tutorial focuses on carrying out that calculation, checking its assumptions, and interpreting the numerical result. The calculation is exact only for the independence model described below; the data used in the examples are hypothetical.

Definition: Let $R_i$ be the event that look $i$ rejects its null hypothesis when the trial-wide null hypothesis $H_0$ is true. The overall Type I error across $K$ looks is $P(R_1 \cup \cdots \cup R_K \mid H_0)$, the probability of at least one false rejection. If the looks are independent and $P(R_i \mid H_0)=\alpha_i$, then $$ \alpha_{\mathrm{overall}}=1-\prod_{i=1}^{K}(1-\alpha_i). $$ If every look has the same Type I error probability $\alpha$, this becomes $\alpha_{\mathrm{overall}}=1-(1-\alpha)^K$.

The formula uses the complement of the event of interest. At look $i$, the probability of no rejection under $H_0$ is $1-\alpha_i$. Independence makes the probability of no rejection at every look the product of those probabilities. Subtracting that product from 1 gives the probability of at least one rejection.

Assumptions to Check Before Calculating

The calculation is a probability statement under a specified model, not a universal rule for every monitoring plan. Its main assumptions are:

A common true-null setting. The trial-wide null hypothesis is true, so a rejection at any look is a Type I error for the trial.
Known per-look error probabilities. At each look, $\alpha_i$ is the probability of rejection conditional on $H_0$. If each test is exactly calibrated at its nominal level, $\alpha_i$ equals that nominal level. For a conservative or discrete test, its actual rejection probability may be smaller.
Independence of rejection events under $H_0$. The events $R_i$ must be independent for the product formula to hold. Independence of test statistics is one way this can occur, but it must be justified by the design or model.
A fixed set of looks and rules. The number of tests and their rejection criteria are specified for the calculation. Changing the schedule or rules based on accumulating results can change the error probability.

The independent-look setting is useful for learning the arithmetic and for designs that genuinely provide independent tests. It should not be assumed just because tests occur at different times. In particular, repeated analyses of cumulative trial data generally reuse observations, so their test statistics are related. The independent formula is not automatically valid for that setting.

Worked Example: Four Independent Looks at Five Percent

Suppose a hypothetical monitoring plan has four independent tests of the same trial-wide null hypothesis. Each test has an exact Type I error probability of $0.05$ under $H_0$. Calculate the probability of at least one false rejection.

For each look, the probability of no rejection under $H_0$ is $1-0.05=0.95$. By independence, the probability of no rejection at all four looks is

P(\text{no rejection at all four looks}\mid H_0) =(0.95)^4=0.81450625.

Therefore,

\alpha_{\mathrm{overall}}=1-(0.95)^4 =1-0.81450625 =0.18549375\approx 0.1855.

The overall Type I error is about $18.55\%$, even though each individual test has a $5\%$ Type I error probability. This is the probability of at least one false rejection across the four tests under the stated independence and null assumptions; it is not the probability that $H_0$ is true.

Calculations With More Looks or Different Levels

With the same per-look level, increasing $K$ adds another opportunity for rejection. The probability of no rejection, $(1-\alpha)^K$, decreases as $K$ increases when $0<\alpha<1$. Consequently, the overall error increases. When the per-look levels differ, use the product of the separate no-rejection probabilities rather than treating them as if they were all the same.

Worked Example: Six Independent Looks at One Percent

A hypothetical design uses six independent tests, each with an exact per-look Type I error probability of $0.01$ under $H_0$. Find the overall Type I error and compare it with the per-look level.

The probability of no rejection at one look is $0.99$. Thus,

\alpha_{\mathrm{overall}}=1-(0.99)^6.

Calculate the product:

(0.99)^6=0.9414801494, \qquad \alpha_{\mathrm{overall}}=1-0.9414801494 =0.0585198506\approx 0.05852.

The overall error is approximately $5.852\%$, which is larger than the $1\%$ error probability at any single look. The six opportunities do not produce a $6\%$ overall error exactly; the specified independent-event calculation gives about $5.852\%$.

Worked Example: Unequal Per-Look Error Probabilities

Suppose four hypothetical independent looks have actual Type I error probabilities $0.02$, $0.03$, $0.04$, and $0.05$, respectively, under the same trial-wide null hypothesis. Calculate the overall probability of at least one false rejection.

The four probabilities of no rejection are $0.98$, $0.97$, $0.96$, and $0.95$. Their product is

0.98\times 0.97\times 0.96\times 0.95 =0.9506\times 0.96\times 0.95 =0.912576\times 0.95 =0.8669472.

Therefore,

\alpha_{\mathrm{overall}} =1-0.8669472 =0.1330528\approx 0.1331.

Under these assumptions, the overall Type I error is about $13.31\%$. The unequal-level formula accounts for each look's own error probability; replacing all four probabilities with one average would not generally reproduce the same product.

Working Backward From a Target

The same relationship can be rearranged when the desired overall error $A$ and the number of independent looks $K$ are specified. If all looks use a common per-look level $\alpha$, set $A=1-(1-\alpha)^K$ and solve for $\alpha$:

\alpha=1-(1-A)^{1/K}.

This gives the common per-look probability that produces the target overall error in the independent-look model. It is not a general prescription for other dependence structures or monitoring procedures.

Worked Example: A Per-Look Level for a Ten Percent Overall Target

A hypothetical design will use five independent looks and seeks an overall Type I error of $0.10$. Find the equal per-look level that achieves this target under the independence assumptions.

Substitute $A=0.10$ and $K=5$:

\alpha=1-(1-0.10)^{1/5} =1-(0.90)^{1/5}.

Since $(0.90)^{1/5}\approx 0.9791484$,

\alpha\approx 1-0.9791484=0.0208516\approx 0.02085.

Thus, the equal per-look Type I error probability is about $2.085\%$. Under five independent tests with this per-look probability, the overall probability of at least one false rejection is $10\%$, up to rounding.

Calculation Checks and Common Pitfalls

A reliable calculation can be organized as a short sequence: identify the trial-wide null and the rejection event, record the actual per-look error probabilities, verify the independence assumption, multiply the no-rejection probabilities, and subtract from 1. Keep extra digits during multiplication and round only the final result. If reporting a percentage, multiply the final probability by 100 and label it as an overall Type I error percentage.

A common mistake is to say that four tests at $0.05$ give an overall error of exactly $4(0.05)=0.20$. That is not the exact result under independence: the probability of no rejection at all four looks is $0.95^4$, so the exact overall probability is $1-0.95^4\approx 0.1855$. The arithmetic sum and the independent-look formula are different calculations. For this tutorial, use the product formula when the independence assumptions hold.

Another error is to apply the formula based only on the number of scheduled tests, without asking how their results are related. If the tests share participants or data, independence may fail; then multiplying their no-rejection probabilities is not justified. Dependence can change the probability of at least one rejection, so a numerical answer from the independent formula would not be a valid overall error calculation for that design.

The one-look case is a useful check: with $K=1$, the equal-level formula gives $1-(1-\alpha)=\alpha$, as it should. For any $K>1$ and $0<\alpha<1$, the result is greater than $\alpha$ and less than 1. If a computed result violates either check, review the formula, multiplication, and interpretation.

Why the Assumption Matters

The independent-look calculation makes explicit that a per-look significance level and an overall trial-wise error rate answer different questions. The first concerns one test; the second concerns whether any of several tests rejects under the trial-wide null. The product calculation is a transparent benchmark for that distinction, but it is only as appropriate as its assumptions.

The principal alternative is not a different arithmetic version of the same formula: it is to calculate or control the error for the dependence structure and testing plan that actually apply. If the looks are dependent, their joint probability of no rejection is needed, and it cannot generally be recovered by multiplying marginal no-rejection probabilities. In practice, diagnose the design before selecting a calculation. The independent formula is useful when independence is defensible, not as a default for all repeated monitoring.

Key takeaway: For $K$ independent tests with equal per-look Type I error probability $\alpha$, the probability of at least one false rejection under $H_0$ is $1-(1-\alpha)^K$. With unequal probabilities, it is $1-\prod_{i=1}^{K}(1-\alpha_i)$. Both expressions require independence of the rejection events under the null.

Check Your Understanding

Answer each question, then compare with the solutions below. Assume exact per-look error probabilities and independence wherever a question specifies independent tests.

Three independent looks each have per-look Type I error probability $0.02$. What is the overall Type I error?
Two independent looks have Type I error probabilities $0.01$ and $0.04$. Calculate the probability of at least one false rejection.
A single test has Type I error probability $0.05$. What is the overall error in the one-look case, and how does that check the general formula?
Four tests each have a per-look level of $0.05$, but they use cumulative data. Can the independent-look formula be used solely because there are four tests? Explain.

Solutions. (1) $1-(1-0.02)^3=1-0.98^3=1-0.941192=0.058808$, or about $5.881\%$. (2) $1-(1-0.01)(1-0.04)=1-(0.99)(0.96)=1-0.9504=0.0496$, or $4.960\%$. (3) For $K=1$, $1-(1-0.05)^1=0.05$, equal to the single test's error probability. (4) No. Reuse of cumulative data means the rejection events may be dependent, and the product formula requires independence; the number of tests alone does not establish that condition.