Interpreting Monitoring Under the Alternative

Monitoring When an Effect Is Real

Repeated monitoring is often discussed in terms of false positives: when \(H_0\) is true, how often will a monitoring rule reject it at least once? But a monitoring plan also has consequences when an effect is real. It can affect the chance that the study detects the effect, and the look at which the first rejection occurs. These are questions about power and detection timing, not Type I error.

The interpretation depends on the whole decision rule: the alternatives being considered, the planned looks, and the rejection boundary at each look. A nominal significance level for one test does not by itself describe either the probability of any rejection across repeated looks or the chance of detecting a specified effect.

Power is not one universal number for a study. It depends on which alternative value is used. A monitoring plan may have different power for a small effect than for a large one. The probability of an early stop is also distinct: it is the probability of rejecting at a particular look, either unconditionally or conditional on not having rejected earlier. These probabilities describe different features of the monitoring rule.

A Two-Look Illustration

To make the arithmetic transparent, first consider a deliberately simplified model. At each of two planned looks, a one-sided standardized test statistic \(Z_j\) is compared with the same boundary \(c=1.645\). Under the null, each statistic has a standard normal distribution. Under a particular alternative, each has a normal distribution with variance 1 and mean \(\delta\). For this illustration only, assume \(Z_1\) and \(Z_2\) are independent. Each look rejects if its statistic exceeds 1.645, and the study stops at the first rejection.

These assumptions are useful for illustrating repeated opportunities, but independence is not generally appropriate for tests based on accumulating data: successive statistics often share observations and are dependent. The calculations below are not a substitute for modeling that dependence in an actual monitoring plan.

Why the Rejection Probability Changes

The general event structure does not require independent looks. Let \(R_j\) be the event that the statistic at look \(j\) crosses its boundary, and let \(K\) be the number of planned looks. The monitoring rule rejects if the union of those events occurs. Under a parameter value \(\theta\), its rejection probability is

When \(\theta\) is an alternative value, this is the power at \(\theta\). When \(\theta\) is a null value, it is the Type I error probability for the rule. The formula involves the joint behavior of all the looks: knowing each look's separate rejection probability is not enough unless their dependence is also known.

A useful derivation partitions the event of any rejection according to the first look at which rejection occurs. Let \(A_j\) be the event that the first rejection is at look \(j\). These events cannot occur together, and their union is the event of at least one rejection. Consequently,

This decomposition explains how an overall detection probability can be reported by timing. In general, the probability at look \(j\) is not simply the probability of crossing its boundary at that look: a rejection at look \(j\) counts as the first rejection only if earlier looks did not reject.

Power Depends on the Alternative

A result described as “the power” is incomplete unless the alternative is specified. In the simple normal illustration, increasing \(\delta\) shifts the distribution of each test statistic farther above the fixed boundary. That raises the per-look rejection probability and, for the unchanged rule, the probability of at least one rejection. Smaller effects are harder to detect and generally have lower power.

When an Added Look Raises Power

There is a precise but limited statement about adding monitoring opportunities. If a new look is added while all the old looks and their boundaries are left unchanged, the original rejection events remain part of the new rule. The event of any rejection under the expanded rule therefore contains the event of any rejection under the original rule. For any fixed parameter value, including a specified alternative, its rejection probability cannot decrease. It may stay the same if the new look never adds a rejection, or increase if some paths reject only at the added look.

This set-inclusion argument does not say that power must rise when the boundaries are recalculated. A design may alter its thresholds to control the overall Type I error probability. The resulting event is not necessarily a superset of the old rejection event, so power can change in either direction. The exact consequences depend on the full rule and the joint distribution across looks; a per-look significance level alone does not determine them.

This distinction helps compare common approaches. A final-look-only test has no earlier stopping opportunity, so its power and timing are defined by that one test. Repeating an unchanged per-look cutoff can raise both the chance of detecting an effect and the chance of a false positive. A prespecified monitoring plan with boundaries chosen for a stated error objective evaluates power under the alternative while keeping the null error probability as a separate design property. These approaches answer related but different questions; none can be compared solely by counting looks.

Diagnostics and Common Misinterpretations

When interpreting or checking a power calculation for repeated monitoring, verify the following before drawing conclusions:

• Check the parameter condition. State whether a probability is calculated under \(H_0\) or under a specified alternative. Calling a null rejection probability “power” is incorrect.
• Check the event being counted. “Reject at look 2” may mean crossing at look 2 regardless of the past, or first rejecting at look 2 after no earlier rejection. Those are not the same event.
• Check dependence. Multiplying probabilities of no rejection across looks assumes independence. Shared observations or accumulated data can make tests dependent, so that product may be wrong.
• Check the full rule. If boundaries change when looks are added, the new rule may not preserve the old rejection events. The simple claim that adding a look cannot reduce power then does not follow.
• Check the alternative and units. Report the effect size or parameter value at which power is evaluated, and describe effects in the outcome's units when applicable. Power at one effect size does not automatically describe another.

A frequent mistake is to say, “More looks increase power, so the probability of a false positive is unchanged.” The first claim is only guaranteed if the old boundaries and rejection opportunities remain unchanged; the second is generally false for that same unadjusted expansion. In the two-look illustration, per-look null rejection is 5%, but the chance of at least one null rejection is 9.75%. Conversely, a plan that modifies thresholds to manage null error cannot be evaluated for power merely by counting the looks.