Unplanned Looks and Analysis Flexibility

Why Unplanned Analyses Matter

A monitoring plan specifies when the data will be examined and which analyses can lead to a decision. In practice, emerging results may prompt an additional meeting, an extra interim calculation, or a change in which analysis is emphasized. These choices can seem minor: the same study, outcome, and nominal threshold are being used. But if each additional analysis can trigger rejection of \(H_0\), it adds another route to a false-positive conclusion.

The issue is not that every extra look will produce a misleading result. It is that, under \(H_0\), the probability of at least one rejection can differ from the probability associated with any single test. As the earlier material on repeated significance testing establishes, this overall probability depends on the number of opportunities and on dependence among the tests. An unplanned analysis makes the actual set of opportunities differ from the set used to justify the stated error rate.

For this discussion, assume that each test has a valid marginal Type I error probability at most \(\alpha\) when its analysis is fixed in advance and \(H_0\) holds. That condition alone does not guarantee that a collection of tests has overall Type I error \(\alpha\). Nor does it guarantee validity when the test, outcome, or time is selected after inspecting results. A procedure needs to account for the full decision process, not merely quote the threshold used at each individual analysis.

What Adding a Look Can and Cannot Do

First consider a simple case: the original plan has \(m\) analyses, and a new analysis is added while all original analyses and their rejection rules are retained. The expanded any-look event contains the original event. This yields an important result without requiring independence.

The probability increases strictly if the added analysis has a positive chance, under \(H_0\), to reject in situations where none of the original analyses rejects. If its rejection event is entirely contained in the original any-look event, the probability need not increase. Thus adding a possible rejection route cannot lower the any-look probability under these conditions, but the amount of change is not determined by simply counting analyses.

The set-inclusion argument has specific limits. It assumes the original analyses and rules remain available and unchanged, and compares the chance of any rejection. If a protocol replaces old rules with new, more conservative boundaries, the simple containment no longer applies. A revised procedure may control the overall error despite having additional analyses. Also, additional analyses can be used for description or safety monitoring without being additional routes to reject \(H_0\); what matters for this argument is whether they can affect the confirmatory decision.

Dependence and Result-Prompted Analyses

In a trial, successive analyses often use overlapping data. The test statistics can therefore be dependent: a favorable interim result may tend to be followed by a favorable later result. That dependence changes the probability of at least one rejection, so the independent-look calculation cannot simply be transferred to cumulative interim analyses. Dependence does not erase the set-inclusion result, however. If an extra rejection opportunity is added without removing the old ones, the any-look probability is still no smaller; dependence determines how much it changes.

An analysis prompted by emerging results raises a further issue. If investigators choose whether to run an analysis, which outcome to emphasize, or when to examine the data in response to what they have seen, the analysis process itself depends on the data. A test that would have a valid p-value at a fixed, prespecified time is not automatically valid after selecting its time or form based on earlier results. The nominal per-analysis level describes a fixed test; it does not, by itself, describe the full adaptive decision procedure.

It is useful to distinguish two scenarios. In one, an extra look is decided in advance, but omitted from the planned error-control calculation. The decision schedule is fixed, yet the stated calibration does not cover all its looks. In the other, the decision to look is made after reviewing accumulating evidence. The schedule is then data-dependent, and both the added opportunity and the selection rule matter. The second scenario cannot generally be evaluated by pretending that the observed look was fixed in advance.

Worked Error Bound for Multiple Added Analyses

When exact dependence is unknown, a general bound can still be informative. If each of \(m\) fixed analyses has null rejection probability at most \(\alpha\), the union bound gives \(P(\bigcup_{j=1}^{m}A_j\mid H_0)\leq \sum_{j=1}^{m}P(A_j\mid H_0)\leq m\alpha\). This is an upper bound, not usually an exact error probability. It requires valid marginal tests for the analyses being counted; it does not solve the selection problem created by data-driven choices.

Recognizing and Managing Flexibility

A practical review should ask what could have triggered a decision, not only how many formal interim meetings were scheduled. Record whether the plan allowed extra looks, which outcomes and populations were eligible for analysis, whether any analysis could stop the trial or support a confirmatory claim, and whether the choice of analysis depended on observed results. A post hoc analysis can be useful for exploration, but it should not silently inherit the error rate of a different prespecified plan.

The main alternative to uncalibrated flexibility is a prospectively specified procedure that covers the analyses and decisions actually intended. A monitoring plan can account for repeated opportunities by using appropriately calibrated boundaries or another justified error-control strategy. The plan should state the timing, the analyses that count, and the decision rules. If the plan changes, the consequences for error control should be assessed and transparently reported. Merely labeling a look “exploratory” does not change how a result was selected; it does help distinguish exploratory findings from confirmatory conclusions when used accurately.

Diagnostics for analysis flexibility are largely documentary and procedural. Compare the dated monitoring plan with meeting records, analysis code, data-access logs, and the final report. Look for analyses that appear only after a favorable trend, unreported interim examinations, or a claimed nominal level that was justified for fewer looks than were actually used. Such discrepancies do not prove that a particular finding is false. They show that the reported error rate may not describe the analysis path that produced it.

Common Mistakes and Limitations

• Reversing the any-look event. Under a rule that rejects when any p-value is below its threshold, a false rejection occurs if at least one test rejects. No rejection occurs only if every included test fails to reject.
• Assuming independence. Repeated cumulative analyses commonly share data. Independence calculations are useful illustrations only when the assumption is justified.
• Equating a per-look level with overall control. A \(0.05\) threshold for each test does not, by itself, establish a \(0.05\) probability of any false rejection across several tests.
• Assuming every extra analysis has the same effect. Added opportunities can be highly redundant or can create a new route to rejection. The exact change depends on the joint distribution and decision rules.
• Calling an analysis unplanned but treating it as confirmatory. If it was not part of the calibration, its nominal p-value alone does not establish the claimed overall error rate for the data-driven process.

These principles concern false-positive risk under \(H_0\). They do not establish that a particular significant result is false, quantify the probability that \(H_0\) is true, or tell us how an extra look affects power under an alternative. Those questions require other information and methods.