Diagnosing Multiplicity Procedures With Joint Simulations

Why Simulate the Joint Procedure?

A multiple-endpoint group sequential design makes decisions from a collection of statistics observed repeatedly over time. The statistics are correlated across endpoints, and statistics from successive looks on the same endpoint are also correlated. Moreover, a rejection can change later decisions under a prespecified multiplicity rule. Checking one endpoint at a time, or checking only a single look, can therefore miss errors in the implemented procedure.

Joint simulation provides an empirical check of the complete decision rule: generate all endpoint statistics at all planned looks, apply the rule in its actual order, and record whether any true null hypothesis is rejected. The resulting estimate is a diagnostic, not a replacement for the design’s theoretical justification. In particular, a simulation under the global null alone does not establish strong familywise error control.

Joint Model and Target Quantity

Suppose there are \(m\) endpoints and \(K\) planned looks. Let \(t_k\) be the information fraction at look \(k\), with \(0<t_1<\cdots<t_K=1\). For clarity, assume that all endpoints are assessed at these same information fractions. Write \(R\) for the prespecified \(m\times m\) correlation matrix between endpoint score processes. It must be positive semidefinite and have diagonal entries equal to 1.

This covariance gives unit variance for each statistic and the usual canonical correlation across looks. It can be generated by accumulating multivariate normal increments: for interval \((t_{k-1},t_k]\), generate an increment with covariance \((t_k-t_{k-1})R\), add it to the cumulative process, then divide each endpoint’s cumulative score by \(\sqrt{t_k}\). Under an alternative, the score increment for endpoint \(j\) has mean \(\theta_j(t_k-t_{k-1})\). This construction retains dependence across both endpoints and looks.

Let \(T\) denote the set of true null hypotheses in a configuration. If \(D_j\) is 1 when the full procedure rejects endpoint \(j\), and 0 otherwise, the familywise error indicator for one simulated trial is \(\mathbf{1}\{\sum_{j\in T}D_j>0\}\). The familywise error rate is the probability of that event. Strong control requires this probability to be no greater than the design’s target \(\alpha\) for every configuration of true and false null hypotheses, under the assumptions of the procedure.

Simulation Algorithm and Monte Carlo Uncertainty

The simulation must encode the actual rule, not just the nominal boundaries in isolation. That includes which statistics are available at each look, the endpoint-specific local boundaries, any permitted alpha propagation after rejection, and the stopping rule. These features should agree with the prespecified design. The simulated trial stops when the rule says it stops; later statistics must not be used to make decisions after that point.

If \(B\) independent trials are simulated and \(X\) have at least one rejection of a true null, the Monte Carlo estimate is \(\hat{q}=X/B\). Its estimated Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/B}\). This standard error describes simulation noise, not uncertainty in the clinical treatment effect. A rough 95% Monte Carlo interval is \(\hat{q}\pm1.96\sqrt{\hat{q}(1-\hat{q})/B}\); for rates near 0 or 1, a binomial interval such as Wilson’s is preferable. When judging a small excess over \(\alpha\), increase \(B\): a point estimate alone cannot distinguish a small design defect from ordinary simulation variation.

Worked Examples

Justifying the Joint Covariance

The covariance used in the simulation follows directly from the accumulated-score construction. Let \(W_j(t)\) be endpoint \(j\)’s cumulative score noise by information time \(t\), with \(\operatorname{Cov}(W_a(s),W_b(t))=R_{ab}\min(s,t)\). The standardized noise at time \(t\) is \(W_j(t)/\sqrt{t}\). Therefore, for \(s\leq t\), \[ \operatorname{Cov}\left(\frac{W_a(s)}{\sqrt{s}},\frac{W_b(t)}{\sqrt{t}}\right) =\frac{R_{ab}s}{\sqrt{s}\sqrt{t}} =R_{ab}\sqrt{\frac{s}{t}}. \] With \(s=t_k\) and \(t=t_l\), this is the stated covariance formula. In particular, for the same endpoint, \(a=b\), the correlation between looks is \(\sqrt{t_k/t_l}\) when \(t_k\leq t_l\). Treating looks as independent would contradict this derivation and could materially change the simulated stopping and rejection pattern.

What a Simulation Can and Cannot Diagnose

A useful validation checks more than the final error-rate estimate. Inspect marginal means and variances under the null, empirical correlations across endpoints and looks, the number of trials stopping at each look, and the frequency of each rejection sequence. Compare those summaries with their expected behavior under the model. Also test the decision code on deliberately simple inputs: for example, statistics well below every boundary should produce no efficacy rejection, while a statistic beyond a valid active boundary should trigger the prescribed decision. These checks can reveal indexing errors, an incorrectly applied look-specific boundary, a sign error, or a rejected endpoint being counted twice.

Strong error control concerns all configurations of true null hypotheses, not only the case where every endpoint is null. Simulate the global null and selected partial-null configurations, including plausible effect sizes on the false nulls that can alter the procedure’s later path. If the number of endpoints is small, configurations can often be enumerated; otherwise, use a reasoned selection of configurations and effect values. Simulation over a finite set of configurations is still not a mathematical proof for all possible configurations and alternatives.

Joint simulation is most useful as a complement to a theoretical argument or a validated design calculation. A valid closed-testing construction or other established procedure supplies the control justification under its assumptions; simulation can uncover a mismatch between that rule and its software implementation, or show that an assumed dependence model has not been represented as intended. By contrast, simulating separate endpoints independently, using only marginal nominal levels, or checking only the final look discards features that may determine the familywise error rate.

The result is also conditional on the simulation model. A multivariate normal model may be a reasonable canonical approximation for the planned statistics, but it does not automatically represent every feature of a real trial. Unequal information timing, non-normal statistics, changing endpoint availability, or a dependence structure that differs from the assumed one may require a more suitable data-generating model. State the model, information schedule, correlation assumptions, alternative configurations, random-number seed, number of replicates, and exact decision rule so another analyst can reproduce the check.

Common Mistake

• Wrong reasoning: “The global-null simulation estimated 4.7%, below 5%, so strong familywise error control has been verified.”
• Correction: The global-null result checks only the configuration in which every hypothesis is true. Strong control also concerns configurations with some false nulls, whose statistics can affect the path of a multiplicity procedure. Simulate relevant partial-null configurations and rely on the procedure’s theoretical justification for coverage beyond the configurations checked.
• Another warning: A simulated estimate of 5.1% is not automatically a demonstrated failure if its Monte Carlo uncertainty is large. Report the uncertainty, increase the number of replicates when needed, and investigate the implementation rather than making a conclusion from the rounded estimate alone.