Diagnostic: Misspecification of the Planning Effect

Why the Planning Effect Needs a Diagnostic

A conditional-power calculation answers a forward-looking question: given the interim result, how likely is the final test to cross its rejection boundary if the future data follow a specified effect model? The answer depends on that specified effect. If the effect used for planning is larger than the true effect, the calculation can make the remaining study look more promising than it is under the truth; if it is smaller, it can understate the chance of success. Either mismatch can affect whether a design continues, stops for futility, or increases its planned information.

The aim here is to isolate this dependence on the effect assumption. The examples use a one-sided normal test with a fixed final boundary. They do not imply that one conditional-power value is the unconditional power of an adaptive design, nor that the planning effect is an estimate of the effect observed at interim.

Model and Conditional-Power Formula

Let \(I_1\) be the information available at the interim analysis and \(I_T\) a candidate total information level, with \(0<I_1<I_T\). Information is measured on a scale for which the standardized score has drift \(\delta\) per square root of information. Under the normal independent-increments model, the score accumulated by information \(I\) has mean \(\delta I\), variance \(I\), and independent increments over disjoint information intervals. Equivalently, write \(S(I)=\delta I+W(I)\), where \(W(I)\) has independent normal increments, mean zero, and increment variance equal to the increment in information. The standardized statistic is \(Z(I)=S(I)/\sqrt I\).

Suppose the observed interim statistic is \(z_1\), so that the accumulated interim score is \(S(I_1)=\sqrt{I_1}z_1\). Let \(c\) be a fixed upper critical value; for a one-sided test of level \(\alpha\), \(c=z_{1-\alpha}\). Conditional power under effect \(\delta\) is the probability of rejecting at the final analysis, conditional on the interim data, if the future increments follow that effect model.

The second argument of \(N\) is the variance. Therefore the conditional standard deviation is \(\sqrt{(I_T-I_1)/I_T}\), and the one-sided conditional power is

Here \(\Phi\) is the standard normal cumulative distribution function. This expression assumes the same fixed \(I_T\) and boundary \(c\) for the conditional calculation. If a design uses a different final boundary, the appropriate boundary must replace \(c\); the formula is not, by itself, a general group-sequential boundary calculation.

How Misspecification Changes the Answer

Holding the interim result and candidate total information fixed, the planning-effect calculation substitutes \(\delta_p\) for \(\delta\). The exact difference between the normal arguments under the planning and true effects is

where \(A(\delta)\) denotes the standardized argument of \(\Phi\) in the conditional-power formula. This follows because the effect-dependent part of the final conditional mean changes by \((\delta_p-\delta_t)(I_T-I_1)/\sqrt{I_T}\); dividing by the conditional standard deviation \(\sqrt{(I_T-I_1)/I_T}\) gives the stated difference. Since \(\Phi\) is increasing, a planning effect above the true effect overstates conditional power for this favorable-direction test, and a planning effect below the truth understates it.

The magnitude of the discrepancy also depends on how much information remains. With little information left, the effect assumption has less opportunity to shift the final statistic. With more information remaining, the shift in the normal argument grows as \(\sqrt{I_T-I_1}\). The resulting change in probability is not constant: it is often most consequential when the conditional-power calculation is near a decision cutoff.

Interpreting the Diagnostic in a Re-Estimation Rule

A conditional-power target is a rule input, not a guarantee of final power. If sample size is increased only when conditional power falls in a specified range, a misspecified effect can move interim results into or out of that range. An overly optimistic planning effect may make a promising-zone rule increase information less often than it would under the true effect; an overly pessimistic assumption may make increases more frequent. The consequences for unconditional power depend on the distribution of interim results, the rule, the maximum information, and the final testing procedure.

A useful diagnostic holds the interim result fixed and recalculates conditional power over a scientifically plausible range of effects. It can also evaluate the re-estimation decision under each effect scenario. These are distinct questions: the first describes sensitivity of the conditional-power number at a given interim result; the second concerns how the rule would classify that result. Assessing overall operating characteristics requires averaging over possible interim outcomes under each true effect, not treating one interim conditional power as overall power.

This comparison is different from substituting the noisy observed interim effect as the assumed future effect. A planning effect is a prespecified scenario value, while an interim estimate is random and can be unusually high or low. The earlier material on regression to the mean explains why an unusually favorable interim estimate should not automatically be treated as the effect that will persist. For the present diagnostic, the essential question is whether conclusions are robust to plausible values of \(\delta_t\), not whether one value can be declared the truth from the interim result alone.

Assumptions, Alternatives, and Failure Modes

The displayed formula is appropriate when the statistic has the stated normal independent-increments model, information is correctly specified, the remaining information is positive, and the final rejection event is \(Z(I_T)>c\). The assumed effect and true-effect scenarios must be on the same scale and have the same direction convention. For example, changing from a treatment-minus-control effect to a control-minus-treatment effect without changing the rejection direction reverses the interpretation.

If the endpoint variance is estimated, if information is misspecified, or if the statistic is far from the normal model, the calculated conditional power can be inaccurate for reasons beyond the planning effect. Those issues require separate checks. Likewise, the formula does not automatically account for a binding maximum sample size, a random final boundary, or a design-specific adjustment to the final test. The correct conditional rejection probability must reflect the procedure actually planned.

The main alternatives serve different purposes. A fixed-effect conditional-power calculation is transparent and reproducible but answers only under its chosen effect. A sensitivity analysis reports the range of answers across plausible effects and makes the assumption visible. An interim-estimate-based calculation uses observed information about the treatment effect but can be unstable, especially after a favorable look. Effect-threshold rules avoid expressing the decision through conditional power, but they still require explicit consideration of the effects under which the design is expected to perform well. None of these choices removes the need to state the target effect and assess plausible departures from it.

A common mistake is to say, “The interim result gives 0.86 conditional power, so the design has 86% power,” without stating the assumed effect. The correction is to report the conditional power with its conditioning data and effect assumption—for example, “conditional power is approximately 0.86 given this interim statistic and assuming \(\delta=0.25\)”—and then show how it changes under other plausible effects. The number is conditional on both the observed interim data and the future-effect model.