Diagnostic: Instability From Noisy Interim Effects

Why an Interim Estimate Can Produce an Unstable Recommendation

An unblinded sample-size re-estimation rule may use the treatment effect observed at an interim analysis as if it were the effect that will persist in the remaining participants. But an interim estimate is a random quantity: if the same trial design were repeated, its value would vary from trial to trial. When the recommendation is a nonlinear function of that estimate, even moderate sampling variability can produce large changes in the planned total sample size.

The diagnostic question is not merely whether the interim estimate is imprecise. It is whether the rule transforms that imprecision into consequential changes, such as frequent jumps to the maximum sample size or substantial variation in recommendations across repeated trials. We will make that question precise using a hypothetical normal-endpoint setting. The calculations isolate noise in the estimated effect; they do not address other sources of instability, such as estimating the variance.

Model and Re-Estimation Rule

Suppose a continuous outcome is measured in two independently randomized groups with equal allocation. The outcome in each group is normally distributed with the same known variance, \(\sigma^2=8\) outcome-units squared. Let \(\delta\) be the true treatment-minus-control mean difference, and let \(\hat{\delta}\) be the corresponding difference in sample means at the interim analysis. Let \(m\) denote the total number of participants observed at that analysis, split equally between groups.

The calculation assumes independent observations, a common known variance, equal allocation, and no missing outcomes or changes in the target population. Under these assumptions, each group has \(m/2\) observations, so the variance of the difference in sample means is \(8/(m/2)+8/(m/2)=32/m\). Thus the interim estimate has the sampling distribution below. We use it as a repeated-sampling model, not as a claim that a particular observed estimate is close to \(\delta\).

For illustration, suppose the planned total sample size is recalculated from the observed positive effect using \(K=251.2\), a fixed design constant in participant-count times outcome-units squared. The recommendation is rounded up to an integer and capped at 160 participants. If the estimated effect is nonpositive, the rule stops for futility instead of applying the positive-effect formula. Assume \(m<160\). For a positive estimate, define:

Here \(n_{\mathrm{rec}}\) is the recommended total sample size, \(\lceil\cdot\rceil\) means round upward to the next integer, and the lower bound \(m\) prevents recommending a total smaller than the number already observed. This simplified plug-in rule is deliberately explicit; the constant and cap are hypothetical. It is not, by itself, a guarantee of a particular final power or type I error rate. Those operating characteristics depend on the full design and must be evaluated for the design in which the rule will be used.

Why Small Estimate Changes Can Matter

Before integer rounding and the cap are applied, the recommendation as a function of a positive estimate \(x\) is \(g(x)=K/x^2\). Differentiating gives \(g'(x)=-2K/x^3\). Dividing the derivative by \(g(x)\) shows that a small absolute change in \(x\) produces an approximate relative change in the recommendation of \(-2\,dx/x\). This is a local approximation: it is most informative when the estimate is positive and away from a rounding or cap threshold.

The inverse-square relationship is the source of the sensitivity. A lower positive estimate leads to a larger required sample size, and the effect is especially pronounced when the estimate is small. Rounding makes the final rule stepwise rather than smooth; the cap prevents the recommendation from increasing without limit, but can also create a pile-up of recommendations at the maximum.

Repeated-Sampling Probability of Hitting the Cap

To assess instability, calculate how often the rule reaches a consequential threshold over repeated trials. Under the rule above, for a positive \(\hat{\delta}\), a recommendation of 160 occurs when the rounded value is at least 160. Since the value is rounded upward first, the condition is \(251.2/\hat{\delta}^{\,2}>159\), not \(251.2/\hat{\delta}^{\,2}\geq160\). Therefore the maximum-size event is:

At the exact boundary \(c\), the rounded value is 159, not 160. Because a normal variable has zero probability of taking any one exact value, including or excluding that endpoint does not change the probability. The nonpositive estimates are handled separately by futility stopping and are not counted as maximum-size recommendations.

More Than One Threshold Can Reveal Instability

A cap probability is useful but incomplete. A rule may rarely reach its maximum while still producing a broad range of large sample-size recommendations. For example, the probability of recommending at least 100 participants can be calculated using the same logic. If \(m<100\), then for positive \(\hat{\delta}\), rounding upward gives a recommendation of at least 100 when \(251.2/\hat{\delta}^{\,2}>99\), with corresponding threshold \(\sqrt{251.2/99}\approx1.5929\) units; if \(m\ge100\), every positive estimate already yields a recommendation of at least 100.

Diagnostics, Alternatives, and Limitations

Useful diagnostics focus on the recommendation as well as the estimate. For a set of plausible true effects and interim sizes, inspect the distribution of recommended sample sizes, the probabilities of reaching operationally important thresholds, and the frequency of futility stops. A histogram of recommendations can reveal a broad spread or a sharp pile-up at the cap. A plot of recommendation against interim estimate makes inverse-square sensitivity and abrupt changes from rounding visible. These checks should use the exact rule, including its rounding, minimum, and cap.

A fixed-size design avoids this particular source of sample-size variation: the total sample size does not depend on an interim effect estimate. Its trade-off is that it cannot respond to interim information. An effect-threshold rule instead changes the recommendation only when an estimate crosses prespecified cutoffs; it can make the rule easier to communicate, but it may still be sensitive to noise near a cutoff. A plug-in rule uses more of the estimate's numerical value, but its inverse-square form can be especially sensitive to small positive values. No alternative removes the need to examine repeated-sampling behavior under plausible conditions.

The calculations here rely on a known variance, normal sampling, equal allocation, and an interim estimate with the stated variance. In practice, variance estimation, unequal allocation, missing data, or departures from the assumed outcome model can change the distribution of \(\hat{\delta}\) and therefore the threshold probabilities. The rule also treats an observed interim effect as an input to a sample-size formula; it does not establish that the effect is the true effect or guarantee the planned power. Finally, an unblinded adaptation must be evaluated as part of the complete inferential design, since its effect on operating characteristics depends on the analysis and adaptation procedure together.

Common Mistakes

• Ignoring the variability of the interim estimate. A single recommendation calculated at the true effect describes only one input value. Repeated-sampling probabilities describe how the rule behaves across possible estimates.
• Applying the cap condition before rounding. For an integer cap of 160 after rounding upward, the unrounded value must be greater than 159 to round to at least 160. Using 160 instead gives the wrong threshold.
• Counting nonpositive estimates as maximum-size recommendations. In the stated rule, nonpositive estimates trigger futility stopping. They must not be added to the probability of a positive-estimate cap event.
• Interpreting a threshold probability as a probability about the true effect. The 25.9% result is the repeated-sampling probability of a specified recommendation under \(\delta=2\), not the probability that the true effect has a particular value.