Why Test a Fixed Rule Under Different Effect Patterns?
A retention rule can look satisfactory in one configuration and behave poorly in another. A threshold may retain few arms when all treatments are ineffective, yet discard a genuinely useful arm when only one treatment works. When several treatments work, the same rule may retain multiple arms, as intended, or fail to distinguish their differing effects. Evaluating a rule under only one effect pattern can conceal these differences.
This tutorial compares a fixed selection rule across four patterns: all arms have null effects, all have equal effects, one arm has a useful effect, or several arms have useful effects. The purpose is to describe how the rule behaves, not to claim that an interim effect estimate is the true effect or to test efficacy hypotheses. Arm-retention summaries make the patterns comparable.
Model, Rule, and Retention Summaries
Consider \(K\) experimental arms compared with a control. For this illustration, larger treatment effects are better. Let \(\theta_j\) be the standardized true effect for arm \(j\), measured in units of the standard error of its interim score. Assume the interim score \(Z_j\) has a normal distribution with known variance one and mean \(\theta_j\). For the numerical examples, assume the arm scores are independent:
These are explicit simplifying assumptions for the examples. In a trial with a shared control, arm scores may be correlated; independence should not be assumed without justification. The rule considered here is to retain arm \(j\) precisely when \(Z_j>1\). The threshold is fixed across arms and scenarios.
Writing \(\Phi\) for the standard normal cumulative distribution function, the marginal retention probability follows directly from the model and the rule:
The expected number retained is the sum of these marginal probabilities:
This expectation does not require independence. Independence is needed, however, to multiply marginal probabilities to obtain joint event probabilities, or to use a binomial distribution when all arms have the same retention probability.
Worked Example: All Arms Have Null Effects
Worked Example: The Null Pattern
Suppose there are three experimental arms and \((\theta_1,\theta_2,\theta_3)=(0,0,0)\). Under the stated normal model, each arm has retention probability \(q=\Phi(-1)\). The three scores are assumed independent, as specified for these examples.
Using the unrounded normal probability in the calculation,
Since the arm scores are independent and have the same retention probability, \(N_R\) has a binomial distribution with three trials and probability \(\Phi(-1)\). Thus,
So even when all three true effects are zero, this rule retains at least one experimental arm about 40.44% of the time under the model. That is a description of selection behavior, not a probability that any null hypothesis is true.
Four Patterns, One Threshold
The rule can now be evaluated under effect patterns chosen to reveal different aspects of its behavior. In the next examples, an effect of \(\theta=1.5\) is designated useful for illustration; this designation is a scenario label, not a universal clinical threshold. All scores use the same variance-one model and the same \(Z_j>1\) rule.
Worked Example: Equal Effects Across Arms
Let three arms all have effect \(\theta_j=1.5\). Each retention probability is \(q=\Phi(1.5-1)=\Phi(0.5)\). Independence makes the retained-arm count binomial.
The expected number retained is calculated from the unrounded probability:
The probability all three arms are retained is likewise calculated without rounding the intermediate probability:
The rule often retains a majority of the three equally useful arms, but the probability of retaining all three is about 33.06%. A low probability of retaining all arms is not automatically a failure: whether retaining every arm is desirable depends on the trial’s objectives and the costs of continuing arms.
Worked Example: One Useful Arm and Two Null Arms
Now let \((\theta_1,\theta_2,\theta_3)=(1.5,0,0)\). Arm 1 is designated useful; arms 2 and 3 have null effects. The probabilities are \(q_1=\Phi(0.5)\) for the useful arm and \(q_2=q_3=\Phi(-1)\) for the null arms. The expected total is
The useful arm is retained with probability \(\Phi(0.5)\approx0.6915\). A more specific event is that the useful arm is retained and neither null arm is retained. Independence gives
Thus the rule retains the designated useful arm in about 69.15% of repetitions, but retains only that arm in about 48.95%. Those are different summaries: the first concerns one arm, while the second also requires both other arms to be dropped.
Worked Example: Several Arms With Different Effects
Suppose the effect pattern is \((1.5,1.2,0)\): two arms have positive effects, one at the designated useful level, and the third is null. Their retention probabilities are \(\Phi(0.5)\), \(\Phi(0.2)\), and \(\Phi(-1)\). By linearity of expectation,
Under the independence assumption, the probability that all three are retained is
A modest expected count does not imply that the rule consistently retains the two positive-effect arms. The arm with \(\theta=1.2\) is retained with probability \(\Phi(0.2)\approx0.5793\), and the null arm is retained with probability \(\Phi(-1)\approx0.1587\). Arm-specific probabilities help reveal this distinction.
How to Compare Sensitivity Profiles
A useful sensitivity profile reports more than a single expected count. For three arms, the following summaries show how the fixed rule behaves in the four illustrative scenarios. Values are calculated from the unrounded normal probabilities and rounded only for display.
| Effect pattern \((\theta_1,\theta_2,\theta_3)\) | Expected retained arms | Retention probabilities by arm | Interpretive feature |
|---|---|---|---|
| \((0,0,0)\) | 0.4760 | 0.1587, 0.1587, 0.1587 | Null-arm retention behavior |
| \((1.5,1.5,1.5)\) | 2.074 | 0.6915, 0.6915, 0.6915 | Retention when effects are equal and useful |
| \((1.5,0,0)\) | 1.009 | 0.6915, 0.1587, 0.1587 | Useful-arm retention versus null-arm retention |
| \((1.5,1.2,0)\) | 1.429 | 0.6915, 0.5793, 0.1587 | Different retention rates for two positive effects and a null effect |
These summaries answer different questions. \(E(N_R)\) describes the average number retained across repeated trials under a scenario. \(P(I_j=1)\) describes how often a particular arm is retained. A joint probability, such as the probability that every useful arm is retained and every null arm is dropped, combines several decisions and depends on their joint distribution. Reporting the expected count alone can hide poor performance for a particular arm.
Derivation: Expected Retained-Arm Count
The expectation formula follows from indicator variables. Since \(N_R=\sum_j I_j\), linearity of expectation gives \(E(N_R)=\sum_j E(I_j)\). An indicator equals one with probability \(q_j\) and zero otherwise, so \(E(I_j)=1q_j+0(1-q_j)=q_j\). Therefore,
No independence assumption appears in this derivation. By contrast, for independent scores the probability that no arm is retained is
If arm scores are dependent, the product generally does not hold. The expectation still equals the sum of marginal retention probabilities, but joint summaries require the dependence structure as well as the marginal distributions.
Diagnostics, Alternatives, and Limitations
A sensitivity profile is informative when it makes undesirable behavior visible. Check whether a rule has an unacceptably high probability of retaining null arms, whether it often drops arms designated useful, and whether its behavior changes sharply when effects differ modestly. In particular, compare the useful-arm retention probability across scenarios rather than inferring it from the total expected count.
A common mistake is to report an average of about one retained arm in the one-useful-arm scenario and conclude that the useful arm is almost always selected. That conclusion does not follow: the average combines the useful arm’s retention with the null arms’ retention. In this example the useful arm is retained with probability about 0.6915, while each null arm is retained with probability about 0.1587; the joint probability of retaining only the useful arm is about 0.4895.
- Do not confuse marginal and joint probabilities. Multiplying marginal probabilities is justified here only under the stated independence assumption.
- Do not interpret a scenario as observed truth. The effect vectors are hypothetical inputs for examining operating behavior, not estimates established by the interim data.
- Check model sensitivity. Variance misspecification, unequal information, and shared-control correlation can change retention probabilities. The independent, variance-one model is a transparent illustration, not a universal MAMS model.
- Separate selection from testing. These summaries describe arm retention. They do not establish efficacy or provide familywise error control for later tests.
The threshold rule is also only one way to select arms. A rank-based rule can force a fixed number of arms to continue, even when every arm has weak evidence; a threshold rule can retain none or several. Conditional-power rules use projected future evidence rather than just the interim score, and simulation can evaluate a rule under a richer joint model. These approaches answer different design questions, so comparisons should use the same effect patterns and clearly stated performance summaries.
Check Your Understanding
Use the model and fixed rule \(Z_j>1\) from this tutorial. Solutions are included for checking each calculation.
- For two null arms with independent scores, what is the expected number retained?
- For two arms both having \(\theta=1.5\), what is the probability both are retained?
- For the pattern \((1.5,0)\), what is the probability the useful arm is retained, regardless of the other arm?
- Does the formula for expected retained-arm count require independent scores? Explain.
Solutions.
1. Each null arm has retention probability \(\Phi(-1)\). Thus \(E(N_R)=2\Phi(-1)=2(0.1586552539)=0.3173105078\), or about 0.3173 arms. Linearity of expectation gives this result even without independence.
2. Independence gives \(P(\text{both retained})=\Phi(0.5)^2\approx0.4781\). The calculation concerns a joint event, so the independence assumption matters.
3. The useful arm has \(\theta=1.5\), so its marginal retention probability is \(\Phi(1.5-1)=\Phi(0.5)\approx0.6915\). No condition on the other arm is imposed.
4. No. The expectation is \(E(N_R)=\sum_j E(I_j)=\sum_j q_j\) by linearity, regardless of dependence. Independence is needed for the product formula for joint retention events.