Simulation Targets for MAMS Designs

Why Simulation Targets Need Precise Definitions

A MAMS design can stop early, drop experimental arms, and make one or more final treatment decisions. Its performance therefore cannot be summarized by a single number. A simulation target is a precisely defined probability or expectation under a specified treatment-effect scenario and a specified design. Changing the scenario or the decision rule can change the target, even when the trial’s planned stages and maximum sample size stay the same.

Let there be $K$ experimental treatments. Write $A$ for the set of treatments declared successful by the design’s final decision rule, $S$ for the set selected for further use or development, and $N$ for the total number of participants randomized over the whole trial. These sets need not be identical: a design might select a treatment without declaring it successful, or declare more than one treatment successful. Their meanings must be fixed by the protocol before targets are calculated.

Definition: An operating characteristic is a probability or expectation describing the behavior of a trial design under a specified scenario for the treatment effects and other data-generating assumptions. Examples include the probability of at least one false success declaration, the probability of detecting a beneficial treatment, the expected total sample size, and the probability that a particular treatment is selected.

The scenario specifies which treatment effects are true and, with the model assumptions, the distribution of the trial data. The protocol specifies how those data lead to interim and final decisions. Consequently, an operating characteristic is a property of the combination of scenario, assumptions, and design—not an unconditional guarantee about the design.

Familywise Error: The Probability of Any False Success

For treatment $k$, let $H_{0k}$ be its null hypothesis of no sufficiently beneficial effect, as defined by the trial. At a particular scenario $\theta$, let $T_0(\theta)$ be the set of treatments for which $H_{0k}$ is true. A false success occurs when at least one such treatment is included in $A$.

Definition: The familywise error rate (FWER) at scenario $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta\{A\cap T_0(\theta)\ne\varnothing\}. $$ It is the probability that the design declares at least one truly null treatment successful.

Under the global null, every experimental treatment is null; the error probability in that scenario is often called the weak FWER. Strong familywise error control requires the FWER to be at most the chosen level $\alpha$ for every scenario in which one or more null hypotheses are true, including partial-null scenarios where some treatments are beneficial and others are null. A low global-null FWER alone does not establish strong control.

Worked Example: Comparing Global-Null and Partial-Null Error

Suppose a hypothetical simulation uses 2,000 independent trial replicates in each of two scenarios. In the global-null scenario, all three treatments are null and 74 replicates declare at least one treatment successful. In a partial-null scenario, treatments 1 and 2 are beneficial, treatment 3 is null, and 61 replicates declare treatment 3 successful.

In the global-null scenario, every success declaration is false if any treatment is declared successful. The estimated FWER is $$ \widehat{\operatorname{FWER}}=\frac{74}{2000}=0.0370. $$

In the partial-null scenario, only a success declaration for treatment 3 counts as a familywise error. The estimated FWER is $$ \widehat{\operatorname{FWER}}=\frac{61}{2000}=0.0305. $$

The larger of these two estimates is 0.0370, but checking just these scenarios does not prove strong control over all possible configurations. Each estimate describes the observed fraction of simulated trials with at least one false success in its own scenario; it is not the probability that the null hypothesis is true.

Power Depends on Which Successes Matter

Power is not a single universal target when several treatments are under study. At a scenario where some treatments truly meet the prespecified beneficial-effect criterion, let $T_1(\theta)$ denote their set. Two useful targets answer different questions. Disjunctive power is the probability that the design declares at least one truly beneficial treatment successful. Conjunctive power is the probability that it declares every treatment in $T_1(\theta)$ successful. The first asks whether the trial finds something effective; the second asks whether it finds all treatments defined as beneficial in that scenario.

Definition: At a scenario with $T_1(\theta)\ne\varnothing$, disjunctive and conjunctive power are $$ P_\theta\{A\cap T_1(\theta)\ne\varnothing\} \quad\text{and}\quad P_\theta\{T_1(\theta)\subseteq A\}, $$ respectively. Treatment-specific power for treatment $k\in T_1(\theta)$ is $P_\theta\{k\in A\}$.

These targets have different interpretations and may differ substantially. Disjunctive power can be high even when the chance of finding every beneficial treatment is modest. Treatment-specific power helps show which arms are more or less likely to succeed. Power is defined relative to the treatment-effect scenario and the success criterion; a result counts as a true success only if it meets that criterion.

Worked Example: Three Power Summaries From One Scenario

Consider 1,200 hypothetical simulated trials with three experimental treatments. In the specified scenario, treatments 1 and 2 meet the beneficial-effect criterion, while treatment 3 is null. Treatment 1 is declared successful in 720 replicates, treatment 2 in 660, and at least one of treatments 1 and 2 is declared successful in 900. Both beneficial treatments are declared successful in 610 replicates.

The disjunctive power estimate is the proportion detecting at least one beneficial treatment: $$ \frac{900}{1200}=0.750. $$ The conjunctive power estimate is the proportion detecting both: $$ \frac{610}{1200}=0.5083. $$ The treatment-specific estimates are $720/1200=0.600$ for treatment 1 and $660/1200=0.550$ for treatment 2. The counts are compatible: the event that both treatments succeed is contained in the event that at least one succeeds. These summaries answer distinct questions and should not be reported simply as “the power” without specifying which target is meant.

Expected Sample Size and Treatment Selection

When interim decisions can stop the trial or remove arms, the number randomized is a random variable. The expected total sample size at scenario $\theta$ is $$ E_\theta(N). $$ It averages total enrollment over repeated trials under that scenario and the specified design. It is not the maximum planned sample size, nor does it say how many participants will be enrolled in a particular trial. If relevant, expected enrollment by arm or stage can also be defined, but each such expectation needs its own precise counting rule.

Selection probabilities describe which treatment or treatments the design selects. For treatment $k$, the probability of selection is $P_\theta\{k\in S\}$. This is not necessarily the probability of a successful declaration, $P_\theta\{k\in A\}$. If the design can select more than one treatment, the probabilities across treatments can sum to more than one; if it can select none, they can sum to less than one. When selection is exactly one treatment or none, the probabilities across treatments sum to the probability that any treatment is selected.

Worked Example: Expected Enrollment and Selection Probabilities

Suppose 1,000 hypothetical trial replicates have the following total enrollment: 200 stop after 180 participants, 600 stop after 300, and 200 continue to 420. Each replicate selects exactly one of two treatments or selects neither. Treatment 1 is selected in 500 replicates, treatment 2 in 350, and neither in 150.

The estimated expected total sample size is $$ \widehat{E(N)} =\frac{200(180)+600(300)+200(420)}{1000} =\frac{300000}{1000} =300\text{ participants}. $$

The estimated selection probabilities are $500/1000=0.500$ for treatment 1, $350/1000=0.350$ for treatment 2, and $150/1000=0.150$ for no selection. The two treatment-selection probabilities sum to 0.850, the estimated probability that some treatment is selected. These are selection summaries, not evidence that either treatment is beneficial; that requires comparison with the scenario’s truth and the success rule.

Why Simulation Estimates Vary

In a simulation, each complete trial replicate produces outcomes such as whether any null treatment was declared successful, whether a beneficial treatment was detected, the total enrollment, and the selected set. Under independent replicates generated from the same scenario and design, a probability target can be estimated by the fraction of replicates in which its defining event occurs. This fraction is an average of zero-one indicators.

Justification: Let $I_r$ equal 1 when event $E$ occurs in replicate $r$, and 0 otherwise. If the $R$ replicates are independent and follow the same scenario, then $E(I_r)=P_\theta(E)$ and $$ \widehat{P_\theta(E)}=\frac{1}{R}\sum_{r=1}^{R} I_r, \qquad \operatorname{Var}\{\widehat{P_\theta(E)}\} =\frac{P_\theta(E)\{1-P_\theta(E)\}}{R}. $$ Thus, the fraction estimates the target without systematic bias under these assumptions, but it varies across simulation runs. Replacing the unknown probability in the variance by its estimated fraction gives the usual estimated Monte Carlo standard error, $$ \widehat{SE}=\sqrt{\frac{\hat p(1-\hat p)}{R}}. $$

For example, in the global-null result above, the estimated Monte Carlo standard error is $$ \sqrt{\frac{0.0370(1-0.0370)}{2000}}=0.00422. $$ This quantifies simulation variability in estimating that scenario-specific FWER; it does not capture uncertainty about whether the assumed data-generating model is appropriate.

Diagnostics, Alternatives, and Limitations

A reported target is hard to interpret or trust unless its event, scenario, and denominator are clear. Useful checks include confirming that the number of simulated replicates used matches the reported denominator, that “any false success” is counted across all truly null treatments, and that the stated power target matches the event actually counted. For selection, check whether multiple selections or no selection are allowed before interpreting probabilities that do not sum to one. For expected sample size, verify that the outcome is total enrollment rather than a stage maximum or an arm-specific count.

Monte Carlo error matters especially for rare events. A small estimated FWER from few replicates may be imprecise; a zero observed count does not establish a zero underlying probability. Increase the number of replicates or report uncertainty when the precision is inadequate for the decision being made. Also examine results across scientifically relevant scenarios: a favorable result at one treatment-effect configuration cannot establish good performance at all others.

Simulation is useful when a MAMS decision rule makes direct calculation difficult, and it can represent complex combinations of stopping and selection. Analytic calculations or exact enumeration can be preferable when available, because they may give a precise answer without Monte Carlo error. Approximate formulas can be quicker for simple settings but may rely on assumptions that do not match the full design. Simulation’s flexibility does not remove the need to specify and justify its assumptions, and it cannot correct a misspecified model or an incorrectly implemented decision rule.

Do not confuse targets: FWER counts false success declarations; selection probability counts selections; power counts true successes under a stated alternative.
Do not infer strong control from one scenario: the global-null FWER does not by itself describe partial-null scenarios.
Do not treat expected enrollment as fixed enrollment: it averages a distribution of possible trial sizes.

Key takeaway: Before evaluating a MAMS design, define each target as an event or expectation under a named scenario. Then keep the decision being counted—false success, true success, selection, or enrollment—explicit in both calculation and interpretation.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a scenario with two null treatments and one beneficial treatment, what event defines a familywise error?
Why can disjunctive power exceed conjunctive power when two treatments are beneficial?
A design selects exactly one treatment or none. Its treatment-selection probabilities are 0.42, 0.31, and 0.12. What is the probability of no selection?
In 500 hypothetical replicates, total enrollment is 200 in 100 trials and 400 in 400 trials. Find the estimated expected total sample size.

Solutions. (1) A familywise error occurs if at least one of the two null treatments is declared successful; the beneficial treatment’s success alone is not an error. (2) Detecting at least one beneficial treatment includes trials that detect only one, whereas detecting both requires both successes, so the latter event is a subset of the former. (3) The probabilities of selecting either treatment sum to $0.42+0.31=0.73$, so the probability of no selection is $1-0.73=0.27$. (4) The estimated expectation is $\{100(200)+400(400)\}/500=180000/500=360$ participants.