Stress-Testing Correlated Outcomes And Unequal Effects

Why Stress-Test the Selection Rule?

A selection rule can look reliable in a balanced example with one common effect and still behave differently when the trial has unequal arm sizes or treatments with genuinely different effects. Shared-control contrasts are also dependent: a favorable fluctuation in the control group affects every contrast in the same direction. Stress-testing makes these features explicit and checks whether the planned selection-and-testing procedure performs as intended across them.

The aim here is not to choose a treatment from one simulated data set. It is to examine a prespecified procedure over repeated hypothetical trials: which arm it selects, how often it detects real effects, and how often it rejects at least one true null hypothesis. The calculations below use three active treatments and a shared control to make these issues concrete.

Model and Dependence from a Shared Control

Suppose patients are independently randomized to a control group and \(m\) active-treatment groups. For this tutorial, assume a continuous endpoint with normally distributed outcomes and a common, known within-arm standard deviation \(\sigma\). The sample mean in group \(j\) is \(\bar{x}_j\), and the treatment–control contrast estimate is \(\hat{\Delta}_j=\bar{x}_j-\bar{x}_C\). Its true effect is \(\Delta_j=\mu_j-\mu_C\). Under these assumptions, \(\hat{\Delta}_j\) has mean \(\Delta_j\) and variance \(\sigma^2(1/n_j+1/n_C)\), where \(n_j\) and \(n_C\) are the treatment and control sample sizes.

The contrast statistics are correlated because their estimates all contain the same control mean. Different treatment-group means are independent, and each is independent of the control mean. Therefore, for two distinct treatments \(i\) and \(j\), the covariance of their contrast estimates is the variance of the shared control mean:

This correlation is positive. With equal treatment-group sizes \(n_T\), it reduces to \(n_T/(n_T+n_C)\). Increasing the control allocation relative to treatment allocations generally reduces the correlation. Unequal treatment sizes produce different correlations for different pairs. The formula assumes a single continuous endpoint with the stated independent-arm model; with multiple correlated endpoints, the endpoint covariance structure must also be represented.

A Selection Rule and Its Multiplicity Protection

One simple procedure selects the treatment with the largest observed contrast and tests that treatment using a Bonferroni-adjusted one-sided p-value. With \(m\) candidate treatments and familywise significance level \(\alpha\), compare each raw p-value with \(\alpha/m\), or report the selected arm’s adjusted p-value \(\min(1,mp)\). In a three-treatment example at \(\alpha=0.05\), the raw-p threshold is \(0.05/3=0.01667\). Bonferroni protection does not require independent tests; this is particularly useful when shared-control contrasts are correlated.

The key guarantee concerns the familywise error rate: the probability of rejecting one or more true null hypotheses. It holds even if only the selected arm is tested, provided every individual p-value is valid under its null and the full family size is used for adjustment.

The second inequality uses validity of each marginal p-value under its true null; it does not assume independent p-values. The argument applies to any subset of true nulls, not only the global null, so the procedure has strong familywise error control under these assumptions. The price is conservatism: Bonferroni does not use the known dependence to improve power.

Stress-Testing Unequal Effects and Allocation

A stress test should vary features that can change either the selection probabilities or the operating characteristics of the tests. Unequal true effects matter because selection is more likely to favor the arm with the largest signal, while unequal sample sizes change standard errors and the correlation matrix. A useful simulation grid includes:

• Null configurations: all treatment effects are zero, and partial-null cases in which some effects are zero while others are positive.
• Effect patterns: equal positive effects, one clearly superior treatment, and modest or nearly tied effects.
• Allocation patterns: balanced arms, a larger control group, and unequal treatment-group sizes, while holding total sample size or other design constraints fixed as appropriate.
• Dependence conditions: the correlation induced by each allocation, plus plausible alternative covariance assumptions if the endpoint or analysis model implies them.

For every simulated trial, apply the complete planned rule, including selection and its multiplicity adjustment. Record the selected arm, whether a true effect was detected, and whether any true null was rejected. Under a partial-null configuration, count a false rejection only when a null treatment is declared efficacious. This separates power to detect non-null treatments from familywise error among the true nulls.

Simulation Diagnostics, Alternatives, and Limits

Monte Carlo estimates have sampling error. If \(B\) independently simulated trials produce an estimated error rate \(\hat{r}\), its approximate Monte Carlo standard error is

For example, an estimated rate near 0.05 from 10,000 replicates has \(SE_{\mathrm{MC}}\approx\sqrt{0.05(0.95)/10000}=0.00218\). A reported rate of 0.052 is therefore not, by itself, persuasive evidence that a procedure exceeds 0.05. When there are no observed errors, the rate is not proven to be zero; a rough 95% upper bound is \(3/B\). Use enough replicates to distinguish the design tolerance from simulation noise, and report the number of replicates alongside the estimated rate.

Warning signs include error-rate estimates that change materially across repeat runs, a partial-null case that was never simulated, or a simulation that selects the arm but omits the prespecified adjustment. Also check that the simulated allocation, endpoint variance, and analysis calculation match the planned trial. A simulation can verify performance only under the data-generating scenarios it represents. It cannot establish robustness to unmodeled variance differences, missing outcomes, or a misspecified endpoint distribution.

These alternatives do not remove the need to test null and alternative configurations that reflect the intended design. A correlation-aware method can lose its advantage if its covariance assumptions are wrong. Conversely, a conservative Bonferroni procedure may be a sensible benchmark when dependence is uncertain. Stress tests should report the assumptions and scenarios, rather than presenting one favorable operating characteristic as universal.

Common Pitfalls

• Assuming shared-control tests are independent. The common control mean induces positive covariance. Calculate or model the correlation rather than treating the test statistics as independent.
• Adjusting only for the selected arm. Selection does not erase the other candidates from the testing family. Use the prespecified family size or a valid selection-adjusted method.
• Testing only the global null. Strong control concerns every configuration of true and false nulls. Include partial-null stress scenarios.
• Interpreting power as selection accuracy. A procedure may detect an effect often without often selecting the treatment with the largest true effect. Track these outcomes separately.
• Over-interpreting a small simulated excess. Compare estimated rates with Monte Carlo uncertainty and the design tolerance before calling a procedure anti-conservative.