Why multiplicity requires a joint reference distribution
When a trial compares several treatments, the analysis may report a conclusion for each pair. Even if every individual test uses significance level \(\alpha\), the probability of making at least one false rejection across the family can exceed \(\alpha\). Adaptive allocation does not remove this multiplicity problem. It changes the reference distribution as well: the probability of a complete allocation path depends on the design’s sequential assignment rule and, potentially, on outcomes observed along the path.
The central task is therefore to calibrate comparisons together under the assignment mechanism that actually generated the data. The maximum-statistic procedure uses the joint distribution of the largest comparison statistic. Closed testing goes further: it tests intersections of hypotheses and provides strong familywise error control when every required intersection test is valid.
Suppose there are \(K\) treatment arms. Let \(H_{ab}\) denote a prespecified null hypothesis comparing arms \(a\) and \(b\), and let \(T_{ab}\) be its test statistic, with larger absolute values indicating greater disagreement with the null. The maximum-statistic approach uses \(M=\max_{a<b}|T_{ab}|\). This is a joint procedure: the reference distribution of \(M\), not a collection of unrelated marginal distributions, determines the rejection threshold.
Replaying adaptive assignment paths
Write the allocation path as \(A=(A_1,\ldots,A_n)\), where \(A_t\) is the arm assigned to participant \(t\). Given the history \(H_{t-1}\) available just before that assignment, the design specifies probabilities \(\pi_t(a\mid H_{t-1})=P(A_t=a\mid H_{t-1})\). The probability of a complete path is the product of its sequential probabilities:
For a randomization test of a sharp null, outcomes under all treatment assignments are specified by that null. In particular, under the global sharp null of no treatment effect, each participant’s outcome is fixed across arms. To construct the reference distribution, replay the design over possible paths: at each step use the assignment probability prescribed by the design, update the history as the design requires, and calculate all comparison statistics for the resulting path. The resulting distribution is weighted by path probabilities, which need not be equal.
A randomization p-value for a statistic \(S\) is the reference probability of obtaining a value at least as extreme as the observed value. For a two-sided maximum statistic, “at least as extreme” means \(M(A)\ge M_{\text{obs}}\). Including ties gives a valid, possibly conservative, test for a discrete reference distribution.
Worked Example: An Exact Maximum-Statistic Test
Consider a hypothetical three-arm trial with two prespecified pairwise comparisons, \(H_{AB}\) and \(H_{AC}\). Under the global sharp null, the design replay gives the following complete-path probabilities and studentized statistics. These values are hypothetical outputs of replaying the stated adaptive design and recalculating the statistics on each path.
| Path | Path probability | \(T_{AB}\) | \(T_{AC}\) | \(M=\max(|T_{AB}|,|T_{AC}|)\) |
|---|---|---|---|---|
| 1 | 0.20 | 2.4 | 1.1 | 2.4 |
| 2 | 0.30 | 1.8 | 2.0 | 2.0 |
| 3 | 0.30 | 0.7 | 2.1 | 2.1 |
| 4 | 0.20 | 0.9 | 0.6 | 0.9 |
The observed path is path 1, so \(M_{\text{obs}}=\max(|2.4|,|1.1|)=2.4\). The exact randomization p-value is
At level \(\alpha=0.05\), the maximum-statistic test does not reject the global null. The joint calculation accounts for the chance of a large statistic in either comparison; it does not treat the two tests as independent. The path weights sum to \(1.00\), as required for a reference distribution.
Closed testing and adjusted p-values
Let \(\mathcal{H}\) be the family of hypotheses to be reported. Closed testing requires a valid level-\(\alpha\) test of every nonempty intersection \(H_I=\bigcap_{j\in I}H_j\), for subsets \(I\) of the family. An individual hypothesis \(H_j\) is rejected only if every intersection containing \(H_j\) is rejected by its local test. This rule yields strong FWER control: if any hypotheses are true, their intersection is true, and rejecting one of them requires rejecting that true intersection.
If \(p_I\) is the p-value from the valid local test of intersection \(H_I\), the closed-testing adjusted p-value for \(H_j\) is
Thus an adjusted p-value is the least permissive result among all intersection tests that must be passed. A maximum-statistic test can serve as a local test when it tests the hypotheses in a particular intersection and its reference distribution is valid under that intersection null. The validity of each local test matters; simply applying the global-null maximum distribution to every intersection is not automatically valid.
Worked Example: Adjusting Three Pairwise Comparisons by Closure
Suppose a hypothetical three-arm trial prespecifies \(H_{AB}\), \(H_{AC}\), and \(H_{BC}\). Assume valid randomization-based local tests have been constructed for all seven nonempty intersections, using the appropriate intersection-specific reference distributions. Suppose their p-values are:
| Tested hypothesis or intersection | Local p-value |
|---|---|
| \(H_{AB}\) | 0.04 |
| \(H_{AC}\) | 0.07 |
| \(H_{BC}\) | 0.12 |
| \(H_{AB}\cap H_{AC}\) | 0.09 |
| \(H_{AB}\cap H_{BC}\) | 0.16 |
| \(H_{AC}\cap H_{BC}\) | 0.18 |
| \(H_{AB}\cap H_{AC}\cap H_{BC}\) | 0.14 |
For \(H_{AB}\), the containing intersections are its singleton, the two pairwise intersections that include it, and the triple intersection. Therefore \(p_{AB,\mathrm{adj}}=\max(0.04,0.09,0.16,0.14)=0.16\). Similarly, \(p_{AC,\mathrm{adj}}=\max(0.07,0.09,0.18,0.14)=0.18\), and \(p_{BC,\mathrm{adj}}=\max(0.12,0.16,0.18,0.14)=0.18\). At \(\alpha=0.05\), none is rejected by closure. In particular, the unadjusted value \(0.04\) for \(H_{AB}\) is not sufficient: that hypothesis cannot be rejected unless every containing intersection passes its local test.
Justification and assumptions
The maximum-statistic test controls error under the null used to generate its joint reference distribution. Under the global sharp null, if the assignment mechanism is correctly replayed, the observed path is distributed according to the same path law used for calibration. Consequently, the probability that \(M_{\text{obs}}\) falls in the rejection region is at most \(\alpha\), with possible conservatism from discreteness.
The same argument supports closed testing only when each local test has level at most \(\alpha\) under its own intersection null. If all hypotheses in an intersection are sharp and the null restrictions determine the outcomes needed to replay the relevant paths, an intersection-specific randomization test may be available. For more general partial intersections, some potential outcomes may remain unspecified. An exact path distribution then cannot be obtained merely by pretending those outcomes are fixed at their observed values. A valid local test may instead require a justified conditioning argument, a conservative procedure valid over the unspecified outcomes, or a different inferential method.
The closed-testing justification is short but important. If at least one hypothesis in the family is true, let \(I_0\) index all the true hypotheses. A false rejection of any member of \(I_0\) requires rejecting the intersection \(H_{I_0}\), because that intersection contains every true hypothesis. Since the local test of \(H_{I_0}\) has size at most \(\alpha\), the probability of any false rejection is at most \(\alpha\). This establishes strong FWER control without assuming that the comparison statistics are independent.
Monte Carlo path generation
When exhaustive enumeration is infeasible, draw complete paths from the design’s sequential assignment law and replay the statistic on each path. For \(B\) simulated paths with \(R\) statistics at least as extreme as observed, a commonly used Monte Carlo randomization p-value is \((R+1)/(B+1)\). This avoids reporting a zero p-value and gives a valid simulation-based test when the observed and simulated paths are exchangeable under the null. Simulation error remains: a value near \(\alpha\) should be reported with adequate precision and, where decisions are consequential, checked with more simulations.
Worked Example: A Monte Carlo Maximum-Test P-Value
In a hypothetical adaptive trial, exhaustive path enumeration is too large. The observed maximum statistic is \(M_{\text{obs}}=2.70\). A researcher simulates \(B=1{,}000\) paths from the null assignment mechanism and obtains \(R=18\) maxima at least as large as 2.70. The corrected estimate is
This is below \(0.05\), so the Monte Carlo maximum test rejects its tested null at the 5% level, subject to the stated simulation procedure and null assumptions. The result is an approximation to the path-tail probability, not an exact enumeration. It also tests only the null for which the simulated paths and outcomes are valid; it does not by itself establish strong FWER control for all partial configurations of effects.
Diagnostics, alternatives, and limitations
Before interpreting results, check that the simulated or enumerated path probabilities reflect the actual adaptive algorithm. In particular, verify the initial assignment rule, probability updates, response timing, any restrictions on assignments, and the rule for handling ties or unavailable arms. For enumerated paths, confirm that their probabilities sum to one. For simulated paths, check that the code reproduces known small examples and that repeated simulations produce stable tail estimates. Statistics must also be recalculated consistently on each path, including any studentization or covariate adjustment used in the observed analysis.
The maximum-statistic procedure is direct and often less conservative than adjusting each comparison without using their dependence. Its straightforward calibration, however, is naturally tied to the null distribution used for the joint maximum. Closed testing can give strong control while allowing different local tests for different intersections, but it can require many tests and careful treatment of partial nulls. A Bonferroni procedure is a simpler alternative: test each of \(m\) hypotheses at \(\alpha/m\), which controls FWER if each marginal test is valid, without requiring their dependence structure. It may be conservative and does not solve invalid randomization calibration.
A common mistake is to generate paths under the global sharp null, use the resulting maximum distribution, and claim strong FWER control automatically. That distribution guarantees control under the global null, when all comparisons are null. Under a partial configuration, the distribution of statistics for true comparisons may change because false comparisons affect observed outcomes, and adaptive assignment can carry those changes forward into later allocation probabilities. Unless a suitable property such as subset pivotality is established for the design and statistics, global-null calibration alone does not prove strong control. Closed testing addresses this issue only when its intersection tests are themselves valid.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why can a global-null maximum-statistic test fail to establish strong FWER control?
- A complete adaptive path has sequential assignment probabilities 0.5, 0.8, and 0.25. What is its probability?
- Three hypotheses have valid closed-testing adjusted p-values 0.03, 0.06, and 0.20. Which are rejected at \(\alpha=0.05\)?
- In a Monte Carlo test, \(B=499\) paths are simulated and \(R=9\) statistics are at least as extreme as observed. Calculate the corrected p-value.
Solutions. (1) Global-null calibration is guaranteed only when all comparisons satisfy that null. Under partial nulls, effects on other arms can alter outcomes and subsequent adaptive assignments, changing the distribution of statistics for true comparisons. Strong control requires valid tests for the relevant intersections or an additional justification. (2) Multiply the sequential probabilities: \(0.5\times0.8\times0.25=0.10\). (3) Reject only the first hypothesis, since \(0.03\le0.05\), while \(0.06\) and \(0.20\) exceed the threshold. (4) The corrected value is \((R+1)/(B+1)=(9+1)/(499+1)=10/500=0.0200\).