Why Subset Pivotality Matters After Adaptation
In a multi-arm trial, a familywise calibration may use the joint distribution of all arm statistics when every treatment effect is null. That shortcut is justified only if the statistics for the truly null arms have the same joint distribution when some other arms are effective. Adaptive sample-size or variance re-estimation can break this property: information from a non-null arm may affect a shared variance estimate, a design choice, or a weight used in a null arm’s test.
This issue is separate from whether adaptation is sensible for power or feasibility. The question here is narrower: does the distribution used to calibrate rejection of true null hypotheses remain correct when other hypotheses are false? The answer cannot be inferred from complete-null performance alone.
For example, write \(T_j\) for the statistic testing arm \(j\)'s null hypothesis. If arms 1 and 2 are both null, the joint law of \((T_1,T_2)\) under the complete null must match its law when arm 3 is effective and arms 1 and 2 remain null. The definition concerns the joint law, not just each statistic's marginal mean or variance. Dependence between statistics matters for procedures calibrated using a maximum or another joint event.
Subset pivotality can allow a single-step maximum-statistic procedure to use complete-null calibration to control error under partial-null configurations. Closed testing also requires valid tests of its intersection hypotheses; a complete-null distribution is not automatically a valid substitute for every intersection distribution. If adaptation changes the law of statistics for true nulls, complete-null calibration alone does not establish strong familywise error control.
A Variance-Re-Estimation Mechanism
Consider a simple, hypothetical three-group setup with equal group sizes \(n\). The independent observations in group \(g\) are normally distributed with mean \(\mu_g\) and common variance \(\sigma^2\). Arm 1 is compared with control, and arm 2 may have a different mean. A variance estimate formed by centering all observations at the overall grand mean includes both within-group variation and between-group differences in means.
This is not the usual pooled within-group variance estimator, which centers observations at their own group means. The distinction matters: the within-group estimator is unaffected in expectation by treatment-mean differences under this model, whereas the grand-mean estimator can absorb treatment effects. A blinded pooled estimate can have this issue because blinding treatment labels does not necessarily remove between-group heterogeneity from the pooled data.
Worked Example: A Pooled Variance Estimate Changes Under a Partial Null
Suppose the common standard deviation is \(\sigma=1\), each of three groups has \(n=12\) observations, and arm 1 is a true null: \(\mu_0=\mu_1=0\). Compare two configurations for arm 2: the complete null \(\mu_2=0\), and a partial-null configuration with \(\mu_2=1.5\). Let \(s_{\mathrm{all}}^2\) be the sum of squared deviations from the grand mean divided by \(3n-1\).
For independent normal observations, the expected sum of squared deviations from the grand mean is \(\((3n-1)\sigma^2+n\sum_g(\mu_g-\bar\mu)^2\)\), where \(\bar\mu\) is the average of the three group means. Therefore
Under the complete null, all group means equal zero, so the between-group term is zero and \(E(s_{\mathrm{all}}^2)=1\). Under the partial null, \(\bar\mu=(0+0+1.5)/3=0.5\), and
so
Thus the variance estimate's expectation changes even though the arm 1 null remains true. A statistic for arm 1 that divides its contrast by this estimate's square root cannot be calibrated by simply treating the denominator as having the same law in both configurations. This calculation diagnoses a source of non-pivotality; it does not claim that every estimator pooled across arms has this defect. Under the stated model, a variance estimate based only on within-group residuals has a different behavior.
The calculation also identifies a practical diagnostic: determine exactly which observations enter the re-estimation and how they are centered. Ask whether a non-null treatment effect contributes to the estimated variance or information. “Pooled” is not enough detail to answer that question.
Adaptive Weights and Shared-Control Dependence
Sample-size adaptation can affect a null statistic even when its stage-specific increments are individually standardized. The key risk is that the adaptation rule is related to interim data that are dependent on the null arm's interim statistic. Shared control observations can create this dependence. A random weight selected using the other arm then changes how much of the null arm's interim statistic appears in the final statistic.
The following stylized normal model makes the mechanism explicit. It is an illustration of naive calibration, not a claim that adaptive combination tests are inherently invalid. Let \(Z_1\) be an interim statistic for arm 1 and \(Y\) an interim statistic for arm 2. Under the null for arm 1, suppose \((Z_1,Y)\) is bivariate normal with unit variances, correlation \(\rho\), and means \(0\) and \(\delta\). The correlation represents dependence such as that induced by a shared control. Let \(E_1\sim N(0,1)\) be an independent future-stage increment. After observing \(Y\), define \(A=1\) if \(Y>0\), and use the combination statistic
The random weight represents a design choice that can arise when adaptation changes the stage information or the combination weights. If \(q_A\) were fixed independently of \(Z_1\), this weighted sum would have a standard normal distribution under the null. Here the choice can be related to \(Z_1\) through \(Y\).
Worked Example: A Null Statistic with an Alternative-Dependent Mean
Use the model above with \(\rho=0.5\). Compare \(\delta=0\), when arm 2 is also null, with \(\delta=1\), when arm 2 is effective and arm 1 remains null. Write \(A=\{Y>0\}\). For a bivariate normal pair, \(E(Z_1\mid Y)=\rho(Y-\delta)\). Hence
where \(\phi\) is the standard normal density; the final equality follows because \(\int_a^\infty u\phi(u)\,du=\phi(a)\) and \(\phi(-\delta)=\phi(\delta)\). Since \(E(E_1)=0\) and \(E(Z_1)=0\), the mean of the adaptive statistic is
For \(\delta=0\), this is \(0.3(0.5)\phi(0)=0.05984\). For \(\delta=1\), it is \(0.3(0.5)\phi(1)=0.03630\). The arm 1 null statistic has different means across the two configurations. Its distribution therefore cannot be the same in both. A threshold calibrated as though the complete-null law applied unchanged to the partial-null case need not have the intended error rate.
The result relies on the stated model: shared-data dependence produces \(\rho\ne0\), the weight depends on \(Y\), and the future increment is independent with mean zero. If the adaptation variable is independent of the null statistic, this particular mean-shift mechanism disappears. Other distributional changes would still need to be checked rather than presumed absent.
Derivation and Conditions for a Safe Calibration
The key expectation calculation follows directly from conditional normality. With standardized bivariate normal variables having correlation \(\rho\), the conditional expectation of \(Z_1\) given \(Y\) is \(\rho(Y-\delta)\), because \(Y\) has mean \(\delta\). Therefore
Set \(U=Y-\delta\), so \(U\sim N(0,1)\) and \(Y>0\) is equivalent to \(U>-\delta\). Since \(\phi'(u)=-u\phi(u)\),
This nonzero truncated expectation is the link between arm 2's effect and arm 1's weighted statistic. More generally, if \(q_A\) takes values \(q_1\) on \(A\) and \(q_0\) otherwise, then
under the independent, mean-zero future-increment assumption. Unless the selection event is independent of \(Z_1\), or the weights coincide, the selected weight can inherit dependence on the true-null statistic. This does not by itself determine a rejection probability; it shows why the assumed null distribution needs fresh justification.
Identify which arm hypotheses are true and which effects may be nonzero. A complete-null analysis alone does not test all configurations relevant to strong control.
Record which interim outcomes, variance estimates, and arm-selection decisions determine sample size, information, or test weights.
For each true-null statistic, assess whether its joint law changes as effects outside the true-null subset vary. Include shared controls and estimated nuisance quantities.
Use a calibration justified for the actual adaptive rule and null configuration. Do not substitute the complete-null distribution without establishing the required invariance.
Diagnostics, Alternatives, and Limitations
Useful diagnostics focus on the mechanism, not just the final rejection rate. Simulate or derive null-statistic distributions while holding a selected arm null and changing the effects in other arms. Compare means, variances, tail probabilities, and joint dependence. For variance adaptation, compare the estimator's distribution across configurations; for sample-size adaptation, inspect the relation between the adaptation variable and each true-null statistic. A visible change is evidence against subset pivotality, though a finite simulation that fails to detect a change is not proof of invariance.
The main alternatives address the problem in different ways:
| Approach | What it requires | What it does not guarantee by itself |
|---|---|---|
| Fixed design with complete-null calibration | Subset pivotality, or another justification that complete-null calibration applies to every relevant subset | Validity after data-dependent re-estimation |
| Intersection-specific calibration | A valid null distribution for each tested intersection under the adaptive design | That a convenient complete-null distribution works for all intersections |
| Conditional-error or combination-test methods | A correctly specified conditional error or combination rule that accounts for adaptation and dependence | Automatic validity merely because the test is called a combination test |
| Bonferroni-style calibration | Valid marginal tests for each true null and an appropriate allocation of error | Correct marginal p-values if adaptation has altered their null laws |
These approaches are not interchangeable without checking assumptions. In particular, independent future increments can support some adaptive procedures, but they do not erase dependence between interim statistics and choices made using those statistics. Conversely, a change in a nuisance estimate does not prove that every possible procedure is invalid: a procedure may explicitly account for the changed distribution or use a variance estimator whose law is invariant to effects outside the null subset.
A limitation of the subset-pivotality check is that it is a distributional condition, not a complete design strategy. Establishing it analytically may be difficult in a complex adaptive design; simulation can reveal problems but cannot certify exact control across all configurations. The relevant family of configurations, nuisance parameters, adaptation rules, and decision boundaries must be specified before claims about strong control can be evaluated.
Common Mistake
Wrong reasoning: “The procedure controls the familywise error rate when every treatment is ineffective, so it controls the familywise error rate whatever the true effects.” This confuses calibration under the complete null with control under every partial-null configuration.
Correction: Strong control requires the probability of rejecting at least one true null to be bounded by \(\alpha\) under every configuration of true and false hypotheses. Complete-null calibration can support that claim only with a valid bridge, such as subset pivotality or a calibration that directly accounts for the adaptive design and the relevant intersections.
Check Your Understanding
Answer each question using the definitions and mechanisms in this tutorial. Brief solutions follow.
- In your own words, what must remain invariant for subset pivotality to hold for a subset \(I\) of true nulls?
- In the three-group variance example, what is \(E(s_{\mathrm{all}}^2)\) when \(n=12\), \(\sigma^2=1\), and group means are \(0,0,1.5\)?
- For the adaptive-weight example, why can \(E[Z_1\mathbf{1}_{\{Y>0\}}]\) be nonzero even though \(E(Z_1)=0\)?
- Does observing that a procedure controls error under the complete null establish strong familywise error control after adaptive re-estimation? Explain.
Solutions. (1) The joint distribution of the statistics for \(I\) must stay the same as effects outside \(I\) vary, with common nuisance parameters fixed. (2) The mean is \(1+12(1.5)/35=1.514\). (3) \(Z_1\) and the selection event are dependent through their correlation with \(Y\); conditioning on \(Y>0\) can select positive values of \(Z_1\), so the truncated expectation need not be zero. (4) No. Strong control concerns every configuration of true and false hypotheses; complete-null calibration needs a valid justification, such as subset pivotality, or a method that accounts for the adaptive design under the relevant configurations.