Why Formalize the Selection Set?
At an interim analysis, a MAMS design uses information collected so far to decide which experimental arms will continue. It is useful to describe this decision separately from the particular statistic or clinical criterion that drives it. The central object is the selection set: the collection of experimental-arm labels retained at that interim analysis.
Formalizing the decision makes its inputs and outputs explicit. It also clarifies which events need probabilities—for example, the event that a particular arm is retained—and helps reveal whether a rule leaves some possible interim data without a defined action. The selection rule is part of the design: it should be specified before the interim outcomes are used to make the decision.
The power set \(2^{\mathcal{E}}\) is the collection of every subset of \(\mathcal{E}\), including the empty set and the full set. Thus, if there are three experimental arms, the possible outputs include \(\varnothing\), each of the three single-arm sets, each of the three two-arm sets, and \(\{1,2,3\}\). The control arm is usually handled separately and continued according to the design’s protocol; it is not included in \(\mathcal{E}\) unless the design explicitly makes it part of the selection decision.
The data \(D\) must be understood as the information actually available under the protocol at the time of the decision. It might include observed outcomes, treatment assignments, or prespecified summaries. This tutorial does not prescribe which interim statistic should be used. The important requirement here is that the inputs to \(\delta\) be stated clearly enough that the output is determined for every possible input.
The Selection Set as a Random Object
Before interim data are observed, \(D\) is random. A deterministic policy therefore induces a random set \(S=\delta(D)\). For each arm \(a\in\mathcal{E}\), define its retention event as \(A_a=\{a\in S\}\). The probability of retaining arm \(a\) is \(P(a\in S)\), evaluated under the joint distribution of the interim data for the design and parameter values in question. The set itself is random; after observing \(D=d\), however, the realized set \(\delta(d)\) is fixed.
This distinction matters because the policy and its operating characteristics are different objects. The policy says what action follows each possible interim data record. Operating characteristics describe how often those actions occur under specified data-generating conditions. The mapping can be stated without choosing a particular model for the outcomes; calculating its probabilities requires such a model or another specified joint distribution.
In a formal probability model, the data \(D\) take values in a sample space \(\mathcal{D}\). For each arm \(a\), the event \(\{d\in\mathcal{D}:a\in\delta(d)\}\) should be measurable, so that its probability is defined. With a finite set of possible data records, this condition is automatic. With continuous data, it is satisfied by ordinary rules defined through measurable comparisons or functions of the data.
Worked Example: Mapping Interim Statuses to a Retained Set
Suppose three experimental arms are labelled A, B, and C. For illustration, suppose the interim record contains a prespecified status for each arm: “continue” or “stop.” The policy retains exactly the arms whose recorded status is “continue.” This example concerns the mapping from statuses to a set; it does not specify how the statuses are determined from outcomes.
The arm set is \(\mathcal{E}=\{\mathrm{A},\mathrm{B},\mathrm{C}\}\). For a record \(d=(c_{\mathrm A},c_{\mathrm B},c_{\mathrm C})\), where each \(c_i\) is either continue or stop, define \(\delta(d)=\{i\in\mathcal{E}:c_i=\text{continue}\}\).
Suppose the observed record is \(d=(\text{continue},\text{stop},\text{continue})\). Substitution into the definition gives \(\delta(d)=\{\mathrm{A},\mathrm{C}\}\). Thus the realized selection set contains A and C, while B is dropped.
If all three statuses are “continue,” the same definition gives \(\delta(d)=\{\mathrm{A},\mathrm{B},\mathrm{C}\}\). If all are “stop,” it gives \(\delta(d)=\varnothing\). Including these cases makes the output well defined even when no experimental arm continues.
Retention Events and Their Probabilities
A useful way to analyze a selection policy is to write arm retention as an indicator. For arm \(a\), let \(I_a=1\) if \(a\in S\), and let \(I_a=0\) otherwise. Then \(I_a\) is a random variable under repeated application of the design, and its expectation equals the probability that arm \(a\) is retained. This follows directly because an indicator is one on its event and zero otherwise.
For a finite set of possible interim records \(d\), the expectation can be expanded as a sum. The indicator in each term contributes one exactly when the policy retains arm \(a\) for that record. This yields a direct calculation from the joint probabilities of the records.
This expression does not require the arms’ interim data to be independent. If several arms share a control group, their interim results may be dependent; the probabilities in the sum must then reflect their joint distribution. The selection set is defined by the policy regardless of dependence, but its probability distribution depends on the distribution of the complete interim data.
Worked Example: Calculating Retention Probabilities From Joint Outcomes
Suppose two experimental arms, A and B, have a finite set of possible interim records \(d_1,d_2,d_3\). Their joint probabilities are \(0.20\), \(0.50\), and \(0.30\), respectively. A prespecified policy maps these records as follows:
| Interim record | Probability | Selection set |
|---|---|---|
| \(d_1\) | 0.20 | \(\{\mathrm{A}\}\) |
| \(d_2\) | 0.50 | \(\{\mathrm{A},\mathrm{B}\}\) |
| \(d_3\) | 0.30 | \(\varnothing\) |
The probabilities sum to \(0.20+0.50+0.30=1.00\), so the records cover the stated distribution. Arm A is retained for \(d_1\) and \(d_2\), so \(P(\mathrm{A}\in S)=0.20+0.50=0.70\). Arm B is retained only for \(d_2\), so \(P(\mathrm{B}\in S)=0.50\). The probability that at least one experimental arm continues is \(0.20+0.50=0.70\), and the probability that exactly A continues is \(0.20\).
These calculations use the probabilities of the joint interim records as given. They do not multiply separate arm-specific probabilities or assume that the two arms’ data are independent. If the records arise from correlated contrasts, that dependence is already represented in their joint probabilities.
Randomized Policies and More Than One Interim Look
A policy can also include deliberate randomization, although many clinical-trial selection policies are deterministic. To represent a randomized policy, let \(U\) be an auxiliary random variable used by the decision rule. The selection set is then \(S=\delta(D,U)\). If the randomization is intended to be independent of trial outcomes, that independence is an assumption to state. Probabilities of retention must then average over both \(D\) and \(U\).
Worked Example: A Randomized Decision Defined by an Auxiliary Variable
Suppose \(\mathcal{E}=\{\mathrm{A},\mathrm{B}\}\), and the observed interim record is of type \(y\). For this type of record, the protocol draws an independent \(U\sim\operatorname{Uniform}(0,1)\). It retains A if \(U\leq0.25\), and otherwise retains B. The rule is \(\delta(y,u)=\{\mathrm{A}\}\) when \(u\leq0.25\), and \(\delta(y,u)=\{\mathrm{B}\}\) when \(u>0.25\).
Because a uniform random variable on \((0,1)\) has probability equal to interval length, \(P(\mathrm{A}\in S\mid D=y)=P(U\leq0.25)=0.25\). Likewise, \(P(\mathrm{B}\in S\mid D=y)=P(U>0.25)=1-0.25=0.75\). The two events are disjoint and cover all possible values of \(U\), so the policy retains exactly one arm with probability \(0.25+0.75=1\). This probability comes from the protocol’s explicit randomization, not from uncertainty about which set the rule prescribes for a given pair \((y,u)\).
For a design with several interim looks, one may use an indexed set \(S_j\) for the set selected at look \(j\). The input to the rule at that look must be stated: it may include the information accumulated by then and any previously selected arms. Formally, one can write \(S_j=\delta_j(D_j)\), or include the earlier history explicitly in the argument of \(\delta_j\). This notation does not imply any particular rule across looks; it simply prevents a later decision from being described as if it depended only on data that were not yet available.
How to Check a Selection Policy
A formal rule is useful only if it is complete and matches the intended protocol. Check the following features before using it to calculate probabilities or evaluate a design:
- Inputs are explicit. State exactly which data and summaries are available at the decision time. Do not leave it unclear whether a quantity is observed, estimated, or supplied by a protocol rule.
- Every possible input has an output. Specify what happens for ties, missing or invalid records, and boundary cases if these can occur. Otherwise the policy may be a partial rule rather than a function.
- Arm labels are handled consistently. Confirm that the output contains only eligible experimental arms and that a data record is matched to the correct arm.
- The empty-set case is addressed. If no experimental arm is retained, state whether the experimental part of the trial stops and how the control is handled.
- Randomization is identified. If the rule uses randomization, specify its distribution and any independence assumptions. Do not treat an unstated random choice as part of a deterministic rule.
- The implemented rule matches the written rule. Evaluate it on test records, including extreme and boundary cases, and verify the resulting arm labels.
The policy should also be distinguished from alternative approaches to describing interim decisions. A verbal description may communicate intent, but a mapping specifies the action for each possible input. A table of cases is often easy to audit for a finite set of inputs; a formula is more compact for a large or continuous data space. Both are equivalent representations when they give the same output for every input. A probabilistic summary such as \(P(a\in S)\) is not a substitute for the policy itself: different policies can have the same marginal retention probability while selecting different sets for particular data records.
Common Pitfall and Limitations
A common mistake is to describe a selection policy only by saying that “the promising arms continue,” then proceed as though the retained set were fully specified. That wording does not define a function: it leaves open what “promising” means, how ties are handled, and what happens when no arm qualifies. The correction is to define the data input and the output set for every possible input, whether by a formula, a decision table, or an equivalent algorithm.
Formalization does not establish that a policy is clinically appropriate, statistically efficient, or well calibrated. Nor does writing \(S=\delta(D)\) determine the distribution of \(S\); that requires a model or other probability specification for \(D\), and for any auxiliary randomization. The benefit is more basic but essential: once the rule is explicit, its consequences can be analyzed without ambiguity. Later design choices determine which summaries and criteria should feed into the mapping.
Check Your Understanding
Answer each question using the definitions in this tutorial. Full solutions follow the questions.
- With \(\mathcal{E}=\{1,2,3\}\), how many possible selection sets are there, including the empty set?
- A policy maps records \(d_1,d_2,d_3\), with probabilities \(0.10,0.60,0.30\), to \(\{1\},\{1,2\},\varnothing\), respectively. Find \(P(1\in S)\), \(P(2\in S)\), and the probability that the selected set is empty.
- Why is \(P(a\in S)\) not generally enough to recover the policy \(\delta\)?
- A policy uses an auxiliary random variable \(U\). What additional information must be stated to define its operating probabilities?
Solutions. (1) A set with three eligible arms has \(2^3=8\) subsets, including \(\varnothing\). (2) Arm 1 is retained for \(d_1\) and \(d_2\), so \(P(1\in S)=0.10+0.60=0.70\). Arm 2 is retained only for \(d_2\), so \(P(2\in S)=0.60\). The empty set is selected for \(d_3\), so its probability is \(0.30\). (3) A marginal retention probability summarizes how often an arm is selected, but it does not identify which set is selected for each possible data record; distinct mappings can have the same marginal probabilities. (4) State the distribution of \(U\), how it enters the rule, and any required dependence or independence relationship between \(U\) and the trial data \(D\).