Sensitivity Analysis for Assumed Effects

Why Vary the Assumed Effect?

Conditional power (CP) asks how likely the planned remaining analysis is to meet its success criterion, given the interim data and an assumption about the treatment effect from now on. That assumption can matter greatly. A plug-in calculation may use the treatment effect estimated at the interim look; a design-effect calculation may instead use the effect specified when planning the trial. If those choices lead to different assessments, a single CP value conceals an important source of uncertainty.

Sensitivity analysis makes that dependence explicit. Rather than asking only for CP at one assumed effect, calculate CP across a scientifically plausible range. The resulting curve helps show whether a conclusion is stable over that range or relies heavily on choosing an optimistic or pessimistic effect. It does not determine which effect is correct, and it does not turn CP into a probability distribution over effects.

Model and Conditional-Power Curve

For a transparent calculation, consider a one-sided test of \(H_0:\theta=0\) against \(H_a:\theta>0\), with a fixed final critical value \(c\). Use the canonical normal model: at information \(I\), the standardized score has variance 1 and mean \(\theta\sqrt{I}\). Here \(\theta\) is the treatment effect in response units, and information is in reciprocal squared response units, so that \(\theta\sqrt{I}\) is dimensionless. The model assumes known or adequately approximated variance, independent information increments, and a correctly specified information schedule.

Let \(I_t\) be the information available at the interim analysis, \(I_T\) the planned final information, and \(Z_t=z_t\) the observed interim statistic. Assume \(0<I_t<I_T\). Conditional on the interim data and a fixed assumed effect \(\theta\), the final statistic has a normal distribution with mean and variance

The first term in the mean carries forward the observed interim score; the second is the assumed mean contribution from the remaining information. For the one-sided final success rule \(Z_T\ge c\), the conditional power curve is therefore

where \(\Phi\) is the standard normal cumulative distribution function. In a group sequential design with additional future looks, the success event is generally crossing one or more prespecified future boundaries. The same sensitivity principle applies, but CP must then be calculated for that full future crossing event; substituting a single final critical value would not represent the planned design.

Why the Curve Increases with the Assumed Effect

For a positive-direction test, the conditional standard deviation does not depend on \(\theta\), while the conditional mean increases with \(\theta\) whenever information remains. This gives a simple justification for the curve's direction. Define \(a=(I_T-I_t)/\sqrt{I_T}\) and \(s=\sqrt{1-I_t/I_T}\). Then \(a>0\), \(s>0\), and \(CP(\theta)=1-\Phi((c-m_0-a\theta)/s)\), where \(m_0=\sqrt{I_t/I_T}\,z_t\). Differentiating yields

where \(\phi\) is the standard normal density. Thus CP is increasing in the assumed beneficial effect under this model. This does not mean every group sequential success probability can be summarized by the final-look formula; it means the effect assumption shifts the distribution of future evidence in the beneficial direction while the specified design is held fixed.

Plug-In and Design-Effect Values on the Same Curve

A plug-in CP calculation inserts an estimated treatment effect, often the interim estimate, into the CP formula as though it were the effect applying to the remaining data. A design-effect calculation inserts the prespecified effect used for planning. These are two evaluations of the same conditional-power function, not two different probability models. They answer different conditional questions because they use different effect assumptions.

In the known-variance canonical model above, a simple interim estimate is \(\hat{\theta}=z_t/\sqrt{I_t}\). This estimate can be noisy, especially when interim information is limited. Plug-in CP treats its value as fixed for the calculation; it does not average over uncertainty about \(\theta\). A predictive probability, by contrast, integrates future success probabilities over a distribution for the unknown effect. That is a distinct quantity.

Assessing Whether a Conclusion Is Robust

Sensitivity analysis is most useful when it is tied to a decision or interpretation. For example, an analysis may label CP below 0.30 as low. That threshold is a decision convention, not a universal statistical boundary. To assess robustness, find whether plausible assumed effects lie on both sides of the threshold, and report the range over which the classification changes. The choice of plausible range should have a scientific rationale; selecting it after seeing the curve can make the result misleading.

Diagnostics, Alternatives, and Limitations

Before interpreting a curve, check that it represents the planned analysis. Verify the interim information fraction, remaining information, observed statistic, direction of the test, and the applicable boundary or boundaries. Under the simple final-look model, the curve should increase with \(\theta\) for a beneficial effect and should remain between 0 and 1. A decreasing curve, values outside that range, or a discontinuity in a smooth one-look calculation signals a sign, scaling, or implementation error. With multiple future looks, the curve may need numerical evaluation of correlated future statistics, but the success event must still match the design.

The canonical calculation also relies on its variance and information assumptions. If the variance is estimated with substantial uncertainty, if information is mismeasured, or if the assumed effect is not an appropriate model for future data, the displayed probabilities can be poorly calibrated. Changing the assumed effect alone does not address those problems. Nor should one silently replace the original group sequential boundaries with a nominal final-look cutoff; that changes the design's rejection rule and can change its error properties.

A sensitivity curve complements, rather than replaces, the main alternatives. A single plug-in value is compact but can be dominated by a noisy interim estimate. A single design-effect value is tied to the planning assumption but does not show how conclusions shift away from it. A predictive probability represents uncertainty about the effect by averaging conditional success probabilities over an effect distribution; it answers a different question and depends on that distribution. Sensitivity analysis instead displays conditional power at fixed effect values, without assigning probabilities to those values.

The essential limitation is that a curve does not choose the scientifically credible effect range or a stopping threshold. Those choices require substantive justification and should be stated. The useful output is not simply a graph: it is a contextual statement identifying the assumptions, range examined, and whether the decision-relevant conclusion changes across that range.