Comparing Enrichment Rules Across Scenario Grids

Why Compare Rules Across Scenarios?

An enrichment rule is a prespecified procedure that uses interim information to decide which population will continue in a trial or support a confirmatory claim. Its performance depends not only on the rule, but also on the conditions that generated the data. A rule may have high power when a biomarker-positive subgroup is common and strongly responsive, yet perform less well when that subgroup is rare or treatment effects are similar across strata. Comparing rules under just one assumed condition can therefore give a falsely reassuring ranking.

A scenario grid makes this dependence visible. It combines plausible values of key design inputs—here, biomarker prevalence and treatment effects in the biomarker-defined strata—and evaluates every candidate rule under the same conditions. The purpose is to describe tradeoffs and identify where conclusions are sensitive, not to hide scenario-specific performance in a single score.

Set Up a Comparable Evaluation

For a binary biomarker, let \(p\) be the prevalence of the positive subgroup, and let \(\delta_+\) and \(\delta_-\) denote the treatment effects in the positive and negative subgroups, respectively, on a specified endpoint scale. Each grid row sets values for \(p,\delta_+,\delta_-\), along with any other inputs needed to generate trial data. A grid should include clinically plausible heterogeneity patterns and, where relevant, null configurations for assessing false-positive behavior.

Before simulating, specify the trial and each rule completely: interim timing, available data, decision boundaries, sample-size limits, final test, and how multiplicity is controlled for the confirmatory claims. The entire adaptive procedure must be evaluated. Reusing a nominal final-test level without accounting for data-dependent population selection can distort error rates; the selection and testing steps must be represented as designed and calibrated. For fair comparison, candidate rules use the same endpoint, recruitment assumptions, and simulation conditions.

Three summaries are particularly useful. Power is the probability that the prespecified confirmatory test rejects its null hypothesis under a stated alternative scenario. Subgroup coverage should have an explicit operational definition. One useful measure is the probability that the final confirmatory population includes a prespecified subgroup of clinical interest. This is a probability across repeated trials, not the percentage of that subgroup in a particular trial. Expected enrollment is the mean number of participants enrolled under a rule; report whether this means randomized participants, screened participants, or both. Screening burden can differ substantially when recruitment is restricted to a less prevalent subgroup.

These measures answer different questions. Power concerns a defined confirmatory claim; subgroup coverage concerns whether a target population remains represented in that claim; enrollment concerns resource use. If a rule can make different claims in different simulations, power must refer to a specified target claim or a clearly stated claim-level success criterion. Otherwise, apparent power differences may simply reflect that rules are testing different hypotheses.

Estimating and Comparing Operating Characteristics

For each rule and scenario, simulate \(M\) independent trial replications and apply the rule exactly as it would be applied in the trial. If \(I_m=1\) when replication \(m\) achieves the specified success criterion and \(I_m=0\) otherwise, estimated power is \(\hat{\pi}=M^{-1}\sum_{m=1}^{M}I_m\). The same indicator form estimates subgroup coverage when success is replaced by the defined coverage event. If \(N_m\) is the enrollment in replication \(m\), estimated expected enrollment is \(\bar{N}=M^{-1}\sum_{m=1}^{M}N_m\).

The justification for the simulation standard error is direct. Under a fixed scenario and correctly implemented independent replications, the success indicators are Bernoulli with success probability \(\pi\), so their variance is \(\pi(1-\pi)\). The variance of their mean is \(\pi(1-\pi)/M\). Replacing the unknown \(\pi\) by \(\hat{\pi}\) gives the estimated Monte Carlo standard error:

This standard error measures simulation noise, not uncertainty about which scenario will occur or uncertainty in the design assumptions. For expected enrollment, the Monte Carlo standard error is the sample standard deviation of the \(N_m\) values divided by \(\sqrt{M}\). If rules are compared using common random numbers—matched simulated patient and trial-level random inputs—the uncertainty in their difference should be estimated from the replication-by-replication differences. Treating matched estimates as independent can misstate that uncertainty.

Diagnostics, Comparisons, and Limitations

Before interpreting a ranking, verify that simulated data follow the stated scenario and that each rule receives only information available at its interim decision. Check that the final testing procedure, including population selection and multiplicity adjustments, matches the intended design. Also inspect whether the number of replications is sufficient: Monte Carlo standard errors should be small relative to differences that could change the design discussion. For enrollment, check the distribution as well as its mean; early stopping or rare long recruitment paths can make the mean alone incomplete.

Compare rules in several complementary ways. Scenario-by-scenario tables show where a ranking reverses. Weighted summaries give a compact comparison when weights are justified and disclosed. Worst-case or range summaries can highlight sensitivity, but they answer a different question from a weighted average and can be dominated by an extreme scenario. Reporting all three views can be useful when the grid is small enough to interpret.

For population-based recruitment calculations, the assumptions connecting prevalence to screening burden matter. For example, if eligible patients have stable prevalence \(p\), screening is independent of biomarker status apart from eligibility, and recruitment continues until \(n_+\) positive patients are identified, the expected number screened is \(n_+/p\). This follows because each screened eligible patient is positive with probability \(p\), so the number screened to obtain \(n_+\) positives has mean \(n_+/p\). That calculation is not appropriate without modification when prevalence changes over time, screening eligibility differs by subgroup, or recruitment has caps and stopping rules; simulate those features directly.

A grid is only as informative as its scenarios. It cannot establish performance under conditions it omits, and a weighted average can be misleading when weights are arbitrary. Simulated operating characteristics also depend on the outcome model, missing-data assumptions, recruitment process, and implementation of the adaptive algorithm. Treat results as conditional comparisons under explicit assumptions, not as guarantees about a future trial.

Common Pitfall

A frequent error is to declare one rule best because it has the highest grid-average power, without stating scenario weights or showing coverage and enrollment. That reasoning treats a chosen summary as if it were an intrinsic property of the rule. The correction is to report all prespecified metrics by scenario, label any weights used for aggregation, and describe the tradeoffs in context. A power gain that occurs alongside reduced subgroup coverage or greater screening burden is not captured by power alone.