Calibrating Intermediate-Outcome Selection Thresholds

Why Calibrate a Selection Threshold?

An intermediate-outcome rule retains an arm when its observed intermediate score is sufficiently favorable. The cutoff \(c\) determines how demanding that rule is: raising \(c\) generally retains fewer arms, while lowering it retains more. Calibration makes the cutoff answer a concrete design question, such as: under a specified model, what cutoff retains 90% of arms whose final effect meets a stated promising-effect criterion?

The answer depends on more than the cutoff. It depends on the scale and direction of the intermediate score, its uncertainty, and its joint distribution with the final effect. A cutoff chosen without those assumptions has no uniquely defined retention probability. Calibration is therefore a model-based design calculation, not a universal recipe for selecting one numerical threshold.

Definition: Let \(F\) denote an arm's final treatment effect, with larger values defined to be more favorable, and let \(S\) be its intermediate selection score, also oriented so that larger values are more favorable. Under the rule “retain if \(S\ge c\),” the retention probability at final effect \(f\) is \(r(c,f)=P(S\ge c\mid F=f)\). For a specified promising set \(\mathcal{P}\), the average retention probability among promising effects is \(P(S\ge c\mid F\in\mathcal{P})\).

A common promising set is \(\mathcal{P}=\{F\ge f_*\}\), where \(f_*\) is a prespecified final-effect boundary. Pointwise calibration chooses \(c\) so that \(r(c,f_*)=q\) at that boundary. Average calibration instead chooses \(c\) so that \(P(S\ge c\mid F\ge f_*)=q\), averaging over a specified distribution of \(F\) above the boundary. Those targets are different: an average target does not require the retention probability to equal \(q\) at every promising effect.

Model and Calibration Equation

For a continuous final effect with density \(f_F\), a useful form of the average-retention equation is

P(S\ge c\mid F\ge f_*)= \frac{\displaystyle\int_{f_*}^{\infty}P(S\ge c\mid F=f)f_F(f)\,df} {\displaystyle P(F\ge f_*)}.

The denominator normalizes the probability over the specified promising set. The numerator weights the conditional retention probability at each final effect by how likely that effect is under the assumed model. Calibration sets this ratio equal to the chosen target \(q\) and solves for \(c\). If the model specifies a conditional normal distribution \(S\mid F=f\sim N(m(f),\sigma_S^2)\), then

P(S\ge c\mid F=f)= 1-\Phi\!\left(\frac{c-m(f)}{\sigma_S}\right) =\Phi\!\left(\frac{m(f)-c}{\sigma_S}\right),

where \(\Phi\) is the standard normal cumulative distribution function. This probability can be substituted into the integral and evaluated numerically.

A Pointwise Calibration

When the design objective concerns a single clinically or scientifically meaningful final effect \(f_*\), the conditional model gives a direct threshold. Suppose \(S\mid F=f\sim N(a+bf,\sigma_S^2)\), with \(b>0\), and require retention probability \(q\) at \(f_*\). Inverting the normal probability gives

q=\Phi\!\left(\frac{a+bf_*-c}{\sigma_S}\right) \quad\Longrightarrow\quad c=a+bf_*-\sigma_S\Phi^{-1}(q).

The positive slope means a larger final effect predicts a larger intermediate score. The normal model and its parameters must be suitable for the score's scale; the formula does not make the model true.

Worked Example: Calibrating Retention at a Boundary Effect

Suppose a hypothetical design uses an intermediate score \(S\), measured in score units, with \(S\mid F=f\sim N(0.20+0.90f,0.60^2)\). Effects \(F\) use a prespecified effect scale, and larger values are favorable. The design objective is to retain 80% of arms with final effect \(f_*=0.60\). Find the pointwise cutoff.

The model's conditional mean at the target effect is \(0.20+0.90(0.60)=0.74\) score units. Since \(\Phi^{-1}(0.80)\approx0.8416\), the cutoff is

c=0.20+0.90(0.60)-0.60(0.8416) =0.74-0.50496 \approx 0.2350\text{ score units}.

Checking by substitution, \(P(S\ge0.2350\mid F=0.60)=\Phi((0.74-0.2350)/0.60)=\Phi(0.8417)\approx0.800\). Thus the calibrated cutoff meets the pointwise 80% target, subject to the stated normal conditional model. At a smaller effect \(f=0.20\), the mean is \(0.20+0.90(0.20)=0.38\), so retention is \(\Phi((0.38-0.2350)/0.60)=\Phi(0.2417)\approx0.595\). Calibration at \(f_*\) does not imply the same retention probability at other effects.

Average Calibration Under a Joint Distribution

An average target requires specifying how final effects are distributed within the promising set. For instance, two designs can share the same boundary \(f_*\) but imply different distributions of effects above it. If one distribution places more probability close to \(f_*\), its average retention may be lower than a distribution that places more weight on large, favorable effects. The threshold must therefore be calibrated to a stated joint distribution, including its distribution of \(F\), rather than to the boundary alone.

Here is a numerical calibration and its integration check. Assume, hypothetically, that \(F\sim N(0,(2/3)^2)\), and that \(S\mid F=f\sim N(0.53762+f,0.8^2)\), where \(S\) is a score in its defined score units. Arms are considered promising when \(F\ge0.50\). A numerical search for a threshold giving approximately 95.1% average retention yields \(c=0.01238\) score units.

Worked Example: Checking an Average-Retention Calibration

Verify the average retention probability for the stated model and cutoff. First, standardize \(F\) by writing \(z=f/(2/3)\). The promising boundary becomes \(z=0.50/(2/3)=0.75\), and \(P(F\ge0.50)=P(Z\ge0.75)\approx0.2266\), where \(Z\sim N(0,1)\).

At \(F=f\), the conditional probability of retention is

P(S\ge0.01238\mid F=f) =\Phi\!\left(\frac{0.53762+f-0.01238}{0.8}\right).

Since \(f=(2/3)z\), the argument of \(\Phi\) is approximately \(0.65655+0.83333z\). The joint probability of being promising and retained is therefore

\int_{0.75}^{\infty}\phi(z)\Phi(0.65655+0.83333z)\,dz \approx 0.2156.

Dividing by the probability of being promising gives

P(S\ge0.01238\mid F\ge0.50) \approx \frac{0.2156}{0.2266} \approx 0.951.

Thus the cutoff gives approximately 95.1% average retention among effects above 0.50 under this joint model. The numerator is not itself the conditional retention probability: it is the probability of both being promising and retained. The denominator is essential.

Why the Calibration Equation Works

For any fixed final effect \(f\), increasing \(c\) cannot increase \(P(S\ge c\mid F=f)\): the event \(S\ge c\) becomes smaller as the cutoff rises. Averaging this conditional probability over a fixed promising set preserves that monotonicity. Under a continuous conditional distribution with positive density over the relevant score range, the average retention probability decreases continuously from 1 toward 0 as \(c\) increases from very low to very high values. Consequently, a target \(q\) strictly between 0 and 1 has a unique cutoff when the average probability changes strictly across that range.

This argument explains a practical numerical method. Define \(g(c)=P(S\ge c\mid F\ge f_*)-q\), evaluate it using the specified model, then find a root of \(g(c)=0\). A bracketing method is often dependable because \(g\) is monotone: choose cutoffs giving opposite signs and narrow the interval until the desired accuracy is reached. Numerical integration error and root-finding error should both be small relative to the precision claimed for the cutoff.

State the target.
Specify whether \(q\) is required at one boundary effect or averaged over a defined promising set.

Specify the joint model.
Give the distributions and parameters for \(F\) and \(S\mid F\), the score direction, and the effect and score scales.

Solve for the cutoff.
Use the pointwise inverse formula when applicable; otherwise solve the conditional-probability equation numerically.

Verify and report.
Substitute the cutoff back into the probability calculation, report the achieved probability, and examine retention across relevant effects.

Diagnostics, Alternatives, and Limitations

A calibrated number should be accompanied by checks that its interpretation matches the design objective. Recompute the probability at the chosen cutoff rather than relying only on the root-finder's status. Check that score orientation is correct: if smaller scores are favorable, the retention event and tail probability must be reversed. Check scale consistency as well; a cutoff in standardized score units cannot be applied directly to a raw score without the corresponding transformation.

Inspect the retention curve \(r(c,f)\) across a range of final effects, not only at the calibration point or as a single average. A high average can conceal weak retention near the promising boundary, while excellent retention at the boundary can coexist with a low average under a different distribution of effects. Sensitivity checks should vary uncertain model parameters and the distribution of \(F\), then repeat the calculation. If small plausible changes substantially move \(c\) or the achieved probability, the design should not present the cutoff as robust.

Approach	What it targets	Main limitation
Pointwise calibration	A chosen retention probability at one final effect, often the promising boundary	Does not control the average over a range of promising effects
Average calibration	A chosen retention probability averaged over a specified promising-effect distribution	Depends on that distribution and can mask low retention for some effects
Fixed, uncalibrated cutoff	A prespecified score value	Has no stated retention guarantee unless evaluated under a model

These approaches answer different questions, and none automatically establishes that a retained arm will succeed on the final endpoint. A retention target is not a final-efficacy probability, a hypothesis-test boundary, or a guarantee about overall trial error rates. Those are separate design properties. Likewise, if several arms use the same nominal cutoff, the per-arm calibration does not by itself determine a joint probability involving all arms.

Common Pitfall

A frequent error is to interpret \(P(S\ge c,F\ge f_*)\) as the retention probability among promising arms. It is a joint probability, so it includes the probability of being promising in the first place. The required conditional probability is

P(S\ge c\mid F\ge f_*)= \frac{P(S\ge c,F\ge f_*)}{P(F\ge f_*)}.

In the numerical example, reporting 0.2156 as the retention rate would confuse the joint probability with the conditional rate; the correct rate is about 0.951 after dividing by 0.2266. A second related mistake is to call an average target a guarantee at every promising effect. It is only an average under the chosen distribution; pointwise checks are needed to understand its behavior near the boundary.

Key takeaway: A selection cutoff is calibrated by defining the retention event, specifying a joint distribution for intermediate and final effects, and solving for the cutoff that achieves the stated pointwise or average probability. The achieved probability is conditional on the model and target set, so verify the calculation and assess sensitivity before interpreting the cutoff.

Check Your Understanding

Use the definitions and methods above. Solutions are included after the questions.

In one sentence, distinguish pointwise calibration at \(f_*\) from average calibration over \(F\ge f_*\).
Suppose \(S\mid F=f\sim N(0.10+0.50f,0.40^2)\). Find the cutoff that gives 90% retention at \(f_*=0.40\).
Why must \(P(S\ge c,F\ge f_*)\) be divided by \(P(F\ge f_*)\) to obtain average retention among promising effects?
If a calibrated average retention probability is 0.90, what can and cannot be concluded about retention at the promising boundary?

Solutions. (1) Pointwise calibration sets \(P(S\ge c\mid F=f_*)\) to the target, whereas average calibration sets \(P(S\ge c\mid F\ge f_*)\) to it under a specified distribution of \(F\). (2) The mean at \(f_*=0.40\) is \(0.10+0.50(0.40)=0.30\). Since \(\Phi^{-1}(0.90)\approx1.2816\), \(c=0.30-0.40(1.2816)=-0.2126\) score units; substitution gives \(\Phi((0.30+0.2126)/0.40)\approx0.900\). (3) The joint probability counts only the fraction of all modeled arms that are both promising and retained. Dividing by the probability of being promising conditions on membership in the promising set. (4) One can conclude that retention averages 90% over the specified distribution above the boundary. One cannot conclude that retention is 90% at the boundary or at every promising effect; those probabilities require separate evaluation.