Deriving False-Drop Probabilities

What Counts as a False Drop?

An intermediate-based retention rule can save resources by stopping arms whose interim results are unpromising. Its cost is that an arm may be dropped even though its final treatment effect is clinically worthwhile. The probability of this event is a measure of the rule’s risk of discarding a worthwhile arm. It is not the probability that a particular observed arm will ultimately succeed, and it is not a familywise error rate.

Let \(\Theta_F\) denote the arm’s true final treatment effect, with larger values assumed to be more beneficial, and let \(\Delta\) be the prespecified minimum effect considered clinically worthwhile. Let \(X\) be the intermediate effect estimate at the selection look. Consider a rule that retains the arm when \(X\geq c\), where \(c\) is the threshold on the intermediate-effect scale. The arm is dropped when \(X<c\). For a specified true final effect \(\theta_F\geq\Delta\), the false-drop probability is \(P(X<c\mid\Theta_F=\theta_F)\). The conditioning matters: the probability is evaluated among cases where the final effect is worthwhile.

Definition: Under an intermediate rule that drops an arm when \(X<c\), the false-drop probability at a fixed worthwhile final effect \(\theta_F\geq\Delta\) is \(P(X<c\mid\Theta_F=\theta_F)\). Under a predictive model for uncertain effects, the false-drop probability among worthwhile effects is \(P(X<c\mid\Theta_F\geq\Delta)\). These are different quantities: the first fixes the final effect, while the second averages over its distribution above the clinical threshold.

Fixed Final Effect Under a Conditional Normal Model

Suppose the intermediate estimate is normally distributed conditional on the true final effect:

X\mid \Theta_F=\theta_F \sim N\!\left(m(\theta_F),\,v(\theta_F)\right).

Here \(m(\theta_F)\) is the modelled mean intermediate estimate and \(v(\theta_F)\) is its conditional variance, including sampling variation at the selection look. This model can encode the expected relationship between intermediate and final effects. The false-drop probability follows by standardizing \(X\):

P(X<c\mid\Theta_F=\theta_F) = \Phi\!\left( \frac{c-m(\theta_F)}{\sqrt{v(\theta_F)}} \right).

This calculation requires the conditional model to be appropriate for the effect and population under consideration, the rule to use the stated threshold, and the variance to represent the uncertainty in the intermediate estimate. A normal approximation may be unsuitable for small samples, highly skewed estimates, or an intermediate measure with substantial floor or ceiling effects. Those issues need assessment before relying on the probability.

Worked Example: False-Drop Risk at a Fixed Worthwhile Effect

Suppose a hypothetical trial defines a final effect of at least \(0.25\) units as clinically worthwhile. At a true final effect of \(\theta_F=0.30\) units, the specified conditional model is \(X\mid\Theta_F=0.30\sim N(0.24,\,0.12^2+0.10^2)\). The first standard deviation, \(0.12\) units, represents conditional variation in the intermediate effect; the intermediate estimate also has a standard error of \(0.10\) units. The rule drops the arm if \(X<0.20\) units.

The conditional standard deviation is \(\sqrt{0.12^2+0.10^2}=\sqrt{0.0244}=0.1562\) units. Thus,

P(X<0.20\mid\Theta_F=0.30) = \Phi\!\left(\frac{0.20-0.24}{0.1562}\right) = \Phi(-0.2561) \approx 0.3989.

Under this model, an arm whose true final effect is \(0.30\) units has about a 39.9% chance of being dropped at the intermediate look. The model’s normality and variance assumptions are stipulated for this hypothetical calculation; in an application they would need justification using the design and relevant data. The result applies at \(0.30\) units, not automatically to every worthwhile effect.

Averaging Over Uncertain Final Effects

Sometimes the aim is to summarize the false-drop probability across a distribution of possible effects, rather than at one fixed value. For example, that distribution may represent uncertainty about effects in a planning model or a predictive distribution across arms. State which interpretation is intended: the resulting probability is conditional on being worthwhile under that distribution, not a frequentist rate over repeated trials at one fixed parameter value.

For a tractable derivation, suppose the joint predictive distribution of the intermediate estimate \(X\) and true final effect \(\Theta_F\) is bivariate normal:

\begin{pmatrix}X\\ \Theta_F\end{pmatrix} \sim N_2\!\left[ \begin{pmatrix}\mu_X\\ \mu_F\end{pmatrix}, \begin{pmatrix} \sigma_X^2 & \rho\sigma_X\sigma_F\\ \rho\sigma_X\sigma_F & \sigma_F^2 \end{pmatrix} \right].

The correlation \(\rho\) describes dependence between \(X\) and \(\Theta_F\) in this joint model. In a predictive analysis it is not automatically the sampling correlation between two estimates. Define \(a=(c-\mu_X)/\sigma_X\) and \(b=(\Delta-\mu_F)/\sigma_F\). Then \(P(X<c)=\Phi(a)\), \(P(\Theta_F<\Delta)=\Phi(b)\), and the joint lower-tail probability is \(\Phi_2(a,b;\rho)\), the bivariate standard normal distribution function with correlation \(\rho\).

The event \(\{X<c,\Theta_F\geq\Delta\}\) consists of all cases below the intermediate threshold except those also below the final-effect threshold. Therefore,

\begin{aligned} P(X<c\mid\Theta_F\geq\Delta) &= \frac{P(X<c,\Theta_F\geq\Delta)} {P(\Theta_F\geq\Delta)}\\ &= \frac{\Phi(a)-\Phi_2(a,b;\rho)} {1-\Phi(b)}. \end{aligned}

The denominator must be positive; otherwise the model assigns no probability to worthwhile effects and the conditional quantity is undefined. This derivation also shows why a marginal probability \(P(X<c)\) is not generally the false-drop probability among worthwhile effects: it includes effects below \(\Delta\) and does not condition on clinical worthwhileness.

Worked Example: Averaging Over a Joint Predictive Model

For a hypothetical planning model, let \(X\) have mean \(0.20\) units and standard deviation \(0.20\) units. Let \(\Theta_F\) have mean \(\Delta=0.25\) units and standard deviation \(0.15\) units, and let their predictive correlation be \(\rho=0.50\). The retention threshold is \(c=0.20\) units. Thus \(a=0\) and \(b=0\), and the model assigns probability \(0.50\) to \(\Theta_F\geq\Delta\).

For standard bivariate normal variables with correlation \(\rho\), the probability that both are below zero is \(\Phi_2(0,0;\rho)=\tfrac14+\frac{\arcsin(\rho)}{2\pi}\). With \(\rho=0.50\), \(\arcsin(0.50)=\pi/6\), so \(\Phi_2(0,0;0.50)=\tfrac14+\tfrac1{12}=\tfrac13\). Substituting,

P(X<0.20\mid\Theta_F\geq0.25) = \frac{0.50-\tfrac13}{0.50} = \frac13 \approx 0.3333.

The model therefore gives a 33.3% false-drop probability among effects at least \(0.25\) units. This is a predictive-model average, not the risk at the single boundary value \(\Theta_F=0.25\). The correlation and marginal distributions are explicit assumptions; changing them can change the result even when the threshold is unchanged.

Thresholds, Precision, and Dependence

The threshold must be expressed on the same scale as the intermediate variable in the model. If a rule retains an arm when an intermediate \(Z\)-statistic is at least \(z_c\), and \(Z=X/s_I\) with known standard error \(s_I\), the equivalent effect-scale threshold is \(c=z_c s_I\). If the standard error varies across arms, then the equivalent \(c\) varies too; a single effect-scale threshold cannot be substituted without accounting for that variation.

Worked Example: Converting a Standardized Threshold

Suppose a hypothetical rule drops an arm when \(Z<0.50\), where \(Z=X/s_I\). For one arm, let \(s_I=0.08\) units, so the equivalent threshold is \(c=0.50(0.08)=0.040\) units. At a fixed worthwhile final effect, suppose the specified conditional distribution is \(X\mid\Theta_F=\theta_F\sim N(0.10,\,0.06^2+0.08^2)\).

The conditional standard deviation is \(\sqrt{0.0036+0.0064}=0.1000\) units. The false-drop probability is

P(Z<0.50\mid\Theta_F=\theta_F) = P(X<0.040\mid\Theta_F=\theta_F) = \Phi\!\left(\frac{0.040-0.100}{0.1000}\right) = \Phi(-0.6000) \approx 0.2743.

The chance of dropping this arm is 27.4% under the stated conditional model. The calculation assumes the intermediate estimate has the specified conditional mean and variance and that the standard error used to define \(Z\) is \(0.08\) units. If the standard error or the modelled relationship differs, the probability must be recalculated.

At a fixed final effect, lowering \(c\) cannot increase \(P(X<c\mid\Theta_F=\theta_F)\); raising \(c\) cannot decrease it. Greater uncertainty in \(X\) can either raise or lower the probability depending on whether its mean is below or above \(c\). Under the joint predictive model, dependence matters because it changes how often low intermediate estimates coincide with worthwhile final effects. Neither a positive nor a negative correlation has a universal direction of impact for every threshold and mean configuration.

Checks, Alternatives, and Limitations

Before interpreting a calculated probability, check that the direction of benefit and the inequality in the rule agree. If larger intermediate effects are favourable, dropping at \(X<c\) is coherent; if smaller values are favourable, the event must be reversed. Also check that units match, that the standard error is for the selection look, and that the joint model’s correlation refers to the quantities actually used in the calculation. A correlation between two noisy observed estimates should not be silently substituted for a correlation between \(X\) and the true final effect.

The main alternatives answer different questions. A fixed-effect calculation reports risk at a stated \(\theta_F\), making it transparent how risk changes across clinically worthwhile effects. A joint predictive calculation averages over worthwhile effects according to a specified distribution, but depends on that distribution and its dependence structure. A direct numerical integration can replace the bivariate-normal formula when a justified joint model is non-normal; it does not remove the need to specify the model. These are alternative summaries, not interchangeable estimates of one universal false-drop rate.

A frequent mistake is to report \(P(X<c)\) as the chance of false dropping. That is only a marginal chance of dropping, including arms with worthwhile and non-worthwhile final effects. Another mistake is to report \(1-P(\Theta_F\geq\Delta\mid X)\) as the false-drop probability. This is the probability of a non-worthwhile final effect given the intermediate result, which conditions in the opposite direction. False dropping asks for the probability of dropping among worthwhile effects, \(P(X<c\mid\Theta_F\geq\Delta)\), or at a fixed worthwhile effect, \(P(X<c\mid\Theta_F=\theta_F)\).

The normal model may misrepresent tail probabilities, particularly if intermediate outcomes are sparse, censored, binary, or affected by treatment-effect heterogeneity. Estimates of the intermediate–final relationship may also be uncertain. A single reported probability can conceal sensitivity to these choices; report the model, threshold, clinical effect definition, and conditioning event alongside it. The calculation concerns a specified rule and model, not a guarantee about an individual arm’s eventual outcome.

Key takeaway: A false-drop probability must specify both the dropping event and what makes the final effect worthwhile. Calculate it at a fixed true effect when that is the target; use a conditional predictive probability only when an explicit joint distribution and averaging population are intended.

Check Your Understanding

Use the definitions and formulas above. Each answer includes the calculation or reasoning needed to check your work.

An arm has a worthwhile fixed final effect, and \(X\mid\Theta_F=\theta_F\sim N(0.30,0.10^2)\). The rule drops when \(X<0.20\). What is the false-drop probability?
In a joint predictive model, the marginal probability of a drop is \(0.40\), the probability of a worthwhile final effect is \(0.50\), and the joint probability of a drop and a non-worthwhile effect is \(0.15\). Find the false-drop probability among worthwhile effects.
A rule retains when \(Z\geq1.25\), with \(Z=X/s_I\) and \(s_I=0.04\) units. What is the effect-scale threshold?
Why is \(P(X<c)\) not generally the false-drop probability among clinically worthwhile effects?
Suppose a model reports \(P(\Theta_F\geq\Delta\mid X\geq c)\). Is this the false-drop probability? State the relevant conditional probability instead.

Solutions. (1) Standardize: \((0.20-0.30)/0.10=-1\), so the probability is \(\Phi(-1)\approx0.1587\), or 15.9%. (2) The joint probability of a drop and a worthwhile effect is \(0.40-0.15=0.25\). Divide by the probability of a worthwhile effect: \(0.25/0.50=0.50\), or 50%. (3) \(c=z_c s_I=1.25(0.04)=0.050\) units. (4) \(P(X<c)\) includes both worthwhile and non-worthwhile effects; the target conditions on worthwhileness, \(P(X<c\mid\Theta_F\geq\Delta)\), or fixes the true effect at a specified worthwhile value. (5) No. It is the probability of a worthwhile effect among arms retained. False dropping is \(P(X<c\mid\Theta_F\geq\Delta)\), or \(P(X<c\mid\Theta_F=\theta_F)\) for a fixed worthwhile effect.