Partial-Null Scenarios for Strong Error Control

Why Simulate Partial-Null Scenarios?

A MAMS design is often first examined when every experimental treatment has no effect. That global-null scenario is useful, but it does not by itself show how the design behaves when some treatments work and others do not. In a multi-arm trial, treatment comparisons may share control data, and interim decisions can affect which comparisons continue. Consequently, the probability of falsely rejecting a particular true null may depend on what is happening in the other arms.

The relevant question is not merely whether a design behaves acceptably when all treatments are ineffective. It is whether, across configurations that include effective treatments, the design keeps the probability of rejecting at least one true null hypothesis within its target. Simulations that include both null and effective treatments are called partial-null scenario assessments when at least one, but not all, treatment hypotheses are true.

For a concrete efficacy setting, let \(\theta_i=\mu_i-\mu_0\) denote the treatment-minus-control mean difference, in outcome points, and use the one-sided hypotheses \(H_i:\theta_i\leq 0\) versus \(H_{a,i}:\theta_i>0\). The true-null set is then \(I_0=\{i:\theta_i\leq 0\}\). Other MAMS applications may use different outcomes or hypotheses; the simulation target is still defined by which hypotheses are true and whether any of them is rejected.

Strong familywise error control means that the probability of rejecting at least one true null is at most the chosen significance level \(\alpha\), for every parameter configuration covered by the design’s control claim. Checking only the global-null configuration, in which all \(K\) null hypotheses are true, addresses weak control: the error probability is controlled when all nulls are true. Strong control is a broader requirement because it also includes configurations with a mixture of true and false nulls.

Model and Simulation Target

The simulation must represent both the data-generating process and the complete decision process of the design. For illustration, suppose individual continuous outcomes in arm \(i\), including control arm \(0\), follow \(Y\sim N(\mu_i,1)\), with outcomes measured in points. Participants’ outcomes are independent conditional on their assigned arms. At each stage, the design uses the accumulated data available at that interim, applies its prespecified stopping and arm-dropping rules, and makes its terminal decisions as specified. Comparisons with a shared control may be dependent; that dependence must be retained in the simulated data rather than removed by simulating each comparison separately.

These assumptions describe an illustrative model, not a requirement that every MAMS trial have normal outcomes or variance 1. For another trial, use an appropriate outcome model, allocation scheme, accrual pattern, and interim-data structure. In all cases, the design’s rules must be fixed before evaluating the simulated decisions. Each replicate represents one whole trial: generate its data, apply the rules at every stage, and record all final rejections. Randomness within a trial may be dependent across comparisons and stages; the repeated simulated trials should be independent draws from the specified trial-generating process.

For a fixed configuration, define \(R_i=1\) if the design rejects \(H_i\), and \(R_i=0\) otherwise. The indicator of a familywise error is \(F=\mathbf{1}\{\text{at least one true null is rejected}\}\). Equivalently, it is 1 if \(\sum_{i\in I_0}R_i\geq 1\), and 0 otherwise. Thus the target is the probability \(P(F=1)\) under that specific configuration—not the probability of rejecting any hypothesis, since rejecting a false null is not a type I error.

Why the Simulated Proportion Targets the Error Probability

For \(B\) independent simulated trials under one fixed configuration, let \(F_b\) be the familywise-error indicator in replicate \(b\). The observed proportion of trials with an error is \(\widehat{q}=B^{-1}\sum_{b=1}^{B}F_b\). Each \(F_b\) is a Bernoulli random variable whose probability of being 1 is the target \(q=P(F=1)\). Therefore,

This justifies using the proportion as a direct simulation estimate of the configuration-specific error probability. The argument does not require the comparisons within each trial to be independent: it uses the error indicator for the whole trial. The proportion is still subject to Monte Carlo variation, so a small difference from \(\alpha\) should not be treated as a definitive conclusion about control.

Covering the True-Null Subsets

With \(K\) treatment hypotheses, there are \(2^K-1\) nonempty possible true-null subsets. A partial-null assessment considers the subsets that are neither empty nor the full set, while the global-null assessment covers the full set. For each subset, it is also necessary to specify the parameter values for treatments outside that subset and any nuisance parameters relevant to the data model. A subset alone does not specify a complete simulation configuration.

The following table illustrates a hypothetical scenario grid for three treatments, with 10,000 trial replicates per row. The entries are simulated counts of replicates with at least one rejection among the listed true nulls. The effective-treatment effects are fixed at selected values for this grid.

Choosing and Diagnosing Scenarios

For a one-sided efficacy hypothesis \(H_i:\theta_i\leq 0\), setting a true-null treatment’s effect at \(\theta_i=0\) examines its boundary. Boundary configurations are often important for error assessment, but one should not automatically assume they are the most adverse in every adaptive, shared-control design. If control across the entire composite null is required, consider other null values or provide a justification that the boundary is least favorable for the design. Also vary relevant nuisance conditions when they could affect interim decisions or the dependence between comparisons.

A practical assessment proceeds by defining a true-null subset, assigning values to all effects and nuisance parameters, simulating complete trials under those values, and counting any rejection in the true-null subset. Repeating this over the selected scenario grid reveals whether an apparently safe global-null result conceals a more challenging partial-null result. The scenario choices and the rationale for them should be recorded so that the scope of the evidence is clear.

Check implementation and results for warning signs. A treatment that is supposed to be null must actually be generated under the null; otherwise, its rejection is not being counted correctly. The true-null set must be determined from the configured parameter values, not from the simulated estimates. The error indicator must include interim and terminal decisions, including decisions after arms are dropped. If the code records only terminal comparisons for arms that survive, it may omit an earlier false rejection or mishandle the design’s decision rules.

How This Assessment Differs From Alternatives

A global-null simulation checks the case in which every treatment null is true. It is an important reference scenario, but it cannot substitute for partial-null checks when strong control is the goal. Conversely, simulating a grid of partial-null scenarios provides empirical evidence only for the configurations represented; a finite grid does not establish control over a continuous parameter space.

An analytic multiplicity procedure can provide a formal control argument when its assumptions apply. Simulation complements such an argument by checking the implemented design under realistic features such as shared controls and interim decisions. Simply simulating each treatment comparison separately is not an adequate substitute: it can discard dependence and selection induced by the actual MAMS design. Nor is the proportion of trials with any rejection the right error measure if some rejected hypotheses are false; only rejections within the true-null set count.

Common Mistakes and Limitations

• Reporting only the global-null result. That assesses one configuration, not mixtures of effective and ineffective treatments. Include partial-null scenarios when assessing strong control.
• Counting every rejection as an error. A rejection of a false null is not a false rejection. Define \(I_0\) from the configured effects and count only rejections within that set.
• Ignoring overlaps. If two true nulls are rejected in one replicate, that replicate contributes one familywise error, not two. Count the union event.
• Assuming the scenario grid is exhaustive. A grid can identify concerning conditions but cannot, by itself, prove a claim over every parameter value. State its scope and justify any broader conclusion.
• Conditioning on surviving an interim. The target is the probability of a false rejection over the complete trial process. Conditioning on an arm surviving changes the question and can misrepresent the unconditional error probability.