Why Use Thresholds on the Effect Estimate?
An unblinded interim analysis gives information about the estimated treatment effect, not just a measure of statistical evidence. A rule written directly in terms of that estimate can therefore be easier to interpret operationally: for example, a protocol can specify the effect-estimate range in which the sample size will increase. The thresholds are design choices, however, not universal cut points. Their meaning depends on the endpoint scale, the assumed variance, the original design, and the conditional-power cutoffs used to define the rule.
This tutorial develops thresholds for a continuous outcome under a normal model with known variance and a one-sided final test. That restricted setting makes the link between an interim effect estimate and conditional power explicit. The rule is illustrative: any actual adaptive design must prespecify its actions and assess the resulting operating characteristics.
Model and Conditional-Power Calculation
Assume independent observations in two groups, a common known standard deviation \(\sigma\), and a constant treatment effect \(\delta=\mu_T-\mu_C\). At the interim analysis there are \(m\) participants per group; the final design has \(N\) per group, with \(N>m\). The interim estimate has standard error \(\sigma\sqrt{2/m}\). For the final one-sided test, take \(H_0:\delta\leq 0\) against \(H_a:\delta>0\), at significance level \(\alpha\), using the fixed critical value \(z_{1-\alpha}\). Assume the future observations are independent of those already observed and follow the same normal model.
The plug-in conditional-power calculation uses \(\hat{\delta}\) as the assumed effect for the future observations as well as the observed interim estimate. It does not claim that the true effect equals \(\hat{\delta}\). It is a planning calculation conditional on the interim data and that plug-in assumption.
Write \(\hat{\delta}_{F}\) for the final difference in sample means. Given the interim estimate, its conditional mean under the plug-in effect is \(\hat{\delta}\), and its conditional variance is \(2\sigma^2(N-m)/N^2\). The final standardized statistic is \(Z_F=\hat{\delta}_{F}/(\sigma\sqrt{2/N})\). Consequently, its conditional mean is \(\hat{\delta}\sqrt{N/(2\sigma^2)}\), and its conditional standard deviation is \(\sqrt{(N-m)/N}\). The estimated conditional power at this final sample size is therefore:
where \(\Phi\) is the standard normal cumulative distribution function. The formula is for a one-sided test and a positive treatment effect; a two-sided test requires the corresponding two-sided rejection boundary and conditional-power calculation.
Deriving an Effect-Estimate Threshold
For a fixed final sample size \(N\), the expression for \(\widehat{CP}\) increases with \(\hat{\delta}\). Thus a conditional-power cutoff \(\gamma\), where \(0<\gamma<1\), corresponds to a unique effect-estimate threshold. Set the conditional-power expression equal to \(\gamma\), apply the inverse normal function, and solve for \(\hat{\delta}\):
Here \(z_{\gamma}\) is the \(\gamma\) quantile of the standard normal distribution. Since the multiplier of \(\hat{\delta}\) in the conditional-power formula is positive, \(\widehat{CP}\geq\gamma\) exactly when \(\hat{\delta}\geq c_{\gamma}\). This is the key translation: a conditional-power cutoff becomes an effect threshold in the original measurement units.
The interim sample size and final sample size enter through the remaining information fraction \(1-m/N\). The threshold also depends on \(\sigma\) and the test boundary. A planning effect \(\delta_0\) enters through the original design choice of \(N\), not by replacing the observed interim estimate in the rule.
Worked Example: Thresholds for a Planned Design
Suppose the endpoint is measured in millimetres, \(\sigma=10\) mm, and the design assumes a treatment effect of \(\delta_0=5\) mm. The planned power is 90% for a one-sided test with \(\alpha=0.025\). Find the theoretical planned sample size per group, then find the interim effect thresholds corresponding to estimated conditional powers of 20% and 80% when \(m=42\) per group.
For a fixed design, the normal-approximation sample-size formula is \(N=2\sigma^2(z_{1-\alpha}+z_{1-\beta})^2/\delta_0^2\), where \(1-\beta=0.90\). Using \(z_{0.975}=1.959964\) and \(z_{0.90}=1.281552\):
Use this theoretical value to show the threshold calculation before any integer sample-size convention is applied. The factor \(\sqrt{2\sigma^2/N}\) is \(\sqrt{200/84.06}\approx1.5425\) mm, and \(\sqrt{1-m/N}=\sqrt{1-42/84.06}\approx0.7074\). For \(\gamma=0.20\), \(z_{\gamma}\approx-0.8416\):
For \(\gamma=0.80\), \(z_{\gamma}\approx0.8416\), giving:
Thus, at the original final sample size, an interim estimate of about 2.105 mm corresponds to plug-in conditional power of 20%, and an estimate of about 3.942 mm corresponds to 80%. These thresholds are below the assumed design effect of 5 mm. That does not mean the design was planned for a 2.105 mm effect: the thresholds mark conditional-power levels at an interim look, while 5 mm was the effect used to plan the original sample size.
Turning Thresholds into a Prespecified Rule
Two thresholds define three regions. For example, a protocol could designate the region between \(c_{0.20}\) and \(c_{0.80}\) as an expansion region. A simple rule might retain the original sample size outside that region and increase to a prespecified cap \(N_{\max}\) inside it. The exact action is a design decision; the thresholds alone do not specify an increase amount or guarantee a particular final power after an increase.
The lower and upper boundaries have a precise interpretation under the calculation above. Below the lower boundary, the estimated conditional power at the original final sample size is below 20%. Between the boundaries, it is at least 20% and at most 80%. Above the upper boundary, it exceeds 80%. A design may choose different conditional-power cutoffs and different actions, provided these are specified in advance and evaluated for the intended design.
Worked Example: Applying a Three-Region Rule
Use the thresholds from the previous example. Consider a hypothetical prespecified rule: keep the original sample size when \(\hat{\delta}<2.105\) mm or \(\hat{\delta}>3.942\) mm; increase to a fixed cap of 120 participants per group when the estimate is between the thresholds, inclusive. Classify interim estimates of 2.0 mm, 3.0 mm, and 4.0 mm.
For \(\hat{\delta}=2.0\) mm, the estimate is below the lower threshold, so the rule retains the original sample size. As a check, the conditional-power normal argument is approximately \((2.0/1.5425-1.959964)/0.7074=-0.94\), giving estimated conditional power of about 17.4%, below 20%.
For \(\hat{\delta}=3.0\) mm, the estimate lies between 2.105 and 3.942 mm, so the rule increases to 120 per group. Its conditional-power argument is approximately \((3.0/1.5425-1.959964)/0.7074=-0.021\), corresponding to conditional power of about 49.2%, which falls in the stated band.
For \(\hat{\delta}=4.0\) mm, the estimate exceeds the upper threshold, so the rule retains the original sample size. Its conditional-power argument is approximately \((4.0/1.5425-1.959964)/0.7074=0.895\), giving conditional power of about 81.5%, above 80%. These decisions follow the specified rule; they do not establish that the 120-per-group expansion is optimal.
How the Assumed Effect and Target Shape the Cutoffs
There are two different targets that must not be confused. The design power target (90% in the first example) helps determine \(N\) under the assumed effect \(\delta_0\). The conditional-power cutoffs (20% and 80% in that example) define thresholds for the interim estimate at that \(N\). Changing \(\delta_0\) while holding the other planning choices fixed generally changes \(N\), which in turn changes the effect thresholds. Changing the conditional-power cutoffs changes the \(z_{\gamma}\) terms directly.
A useful check is to evaluate the conditional power at the assumed effect rather than treating that effect as a threshold. In the example, plugging \(\hat{\delta}=5\) mm into the conditional-power formula gives an argument of approximately \((5/1.5425-1.959964)/0.7074=1.81\), or conditional power of about 96.5%. This is a conditional calculation at a particular interim estimate; it is not the original unconditional 90% design power.
Worked Example: Comparing a Threshold with the Planning Effect
In the same design, suppose the observed interim estimate is 2.5 mm. Determine whether it falls in the expansion region, and explain how it compares with the assumed planning effect of 5 mm.
The estimate satisfies \(2.105\leq2.5\leq3.942\) mm, so it is in the specified expansion region. Its conditional-power argument is approximately \((2.5/1.5425-1.959964)/0.7074=-0.48\), giving an estimated conditional power of about 31.6%. Under the illustrative rule, the sample size is increased to 120 per group.
The estimate is also below the 5 mm planning effect, but that comparison alone does not determine the action. The rule uses the cutoffs derived from conditional-power levels at the original final sample size. In particular, it is not correct to say that every estimate below the planning effect triggers an increase: the 2.0 mm estimate was below the planning effect too, but lay below the lower threshold and did not trigger the specified expansion.
Diagnostics, Alternatives, and Limitations
Before using an effect-threshold rule, check that the sign and units of \(\hat{\delta}\) match the direction of the test, that the interim and final sample sizes use the same per-group convention, and that the variance and normal-model assumptions are appropriate. Recalculate conditional power at each proposed boundary: the lower boundary should give the lower cutoff, and the upper boundary should give the upper cutoff. If numerical calculations do not show that ordering, check the quantiles, remaining-information fraction, and direction of the inequality.
An effect-scale rule is closely related to a Z-statistic-based rule, but the two express the same interim information differently when the standard error is fixed. Since \(Z_I=\hat{\delta}/(\sigma\sqrt{2/m})\), an effect threshold \(c\) corresponds to a Z threshold \(c/(\sigma\sqrt{2/m})\). The effect scale is often easier to communicate in outcome units; the Z scale expresses the estimate relative to its interim standard error. If the standard error is not fixed or known, this simple conversion does not by itself address how variance is estimated.
| Rule scale | What the cutoff represents | Main practical consideration |
|---|---|---|
| Effect estimate | A treatment-effect value in the endpoint's units | Directly interpretable, but depends on the variance and design assumptions used to calibrate it |
| Z statistic | An effect estimate divided by its standard error | Standardizes evidence, but its relation to an effect in endpoint units depends on the standard error |
| Conditional power | A model-based probability of final rejection conditional on interim data and future-effect assumptions | Provides a probability scale, but depends on the chosen effect assumption and final testing rule |
The calculation here assumes known \(\sigma\), a single interim look at a fixed \(m\), independent normal outcomes, and a fixed one-sided final boundary. It uses a plug-in assumption for the future effect. A noisy interim estimate can therefore move across a threshold even when the underlying effect is unchanged. Also, an unblinded sample-size adaptation does not automatically preserve the type I error rate of a fixed design. The complete adaptive testing procedure and its operating characteristics must be assessed under the null and relevant alternatives; a threshold formula alone is not that assessment.
Common Mistake
Wrong reasoning: “The assumed design effect is 5 mm, so any interim estimate below 5 mm must trigger an increase.” This confuses the effect used to plan the original sample size with the boundaries that define the adaptation rule. The planning effect helps set \(N\); the conditional-power cutoffs and interim information determine \(c_{\gamma}\). In the example, the lower threshold is about 2.105 mm, not 5 mm, and the rule assigns different actions to estimates of 2.0 mm and 2.5 mm even though both are below the assumed effect.
Check Your Understanding
Answer each question using the one-sided normal model and threshold definition in this tutorial.
- If the interim estimate equals \(c_{\gamma}\), what is its plug-in conditional power at the specified final sample size?
- In the threshold formula, what happens to the threshold calculation if the remaining-information fraction \(1-m/N\) is zero? Why is that an edge case rather than a usual interim analysis?
- Why does a conditional-power cutoff of 80% correspond to a higher effect threshold than a cutoff of 20%?
- Explain why an estimate below the planning effect does not, by itself, determine whether the sample size is increased.
- If the interim standard error is fixed, how is an effect threshold \(c\) converted to a Z-statistic threshold?
Solutions. (1) By construction, \(\widehat{CP}(c_{\gamma};N)=\gamma\). (2) The term \(\sqrt{1-m/N}\) is zero when \(m=N\), so no future observations remain; the conditional distribution has no future sampling variation. That is not a usual interim analysis because the final sample size has already been reached. (3) Since \(z_{0.80}>z_{0.20}\) and the multiplier \(\sqrt{1-m/N}\) is positive when \(N>m\), the formula gives \(c_{0.80}>c_{0.20}\). (4) The planning effect is used to set the original sample size, whereas the action is determined by the estimate's position relative to the rule's prespecified thresholds. (5) Since \(Z_I=\hat{\delta}/SE\), the corresponding Z threshold is \(c/SE\), with \(SE=\sigma\sqrt{2/m}\) in this model.