Simulation Operating Characteristics For Enrichment Designs

Why Simulate an Enrichment Design?

An enrichment design makes a population decision using trial data: for example, it may continue in a biomarker-positive subgroup when interim evidence suggests a larger treatment effect there. Its performance cannot be summarized by power alone. The design may select the subgroup often but confirm benefit rarely, or it may have acceptable overall success while rarely making a useful all-comer claim. Simulation evaluates the whole decision rule, including selection, the final test, and the number of participants required.

This tutorial develops a simple two-stage design to make those quantities explicit. Its numerical results are hypothetical. The design is an illustration of how to define and evaluate operating characteristics, not a universal recommendation for an enrichment trial.

Model and Decision Rule

Definition: Let \(M=1\) denote biomarker-positive status and \(M=0\) biomarker-negative status. The prevalence is \(\pi=P(M=1)\). Let \(\delta_1\) and \(\delta_0\) be the treatment-minus-control mean differences in the two strata, measured in outcome units; the all-comer mean difference is \(\delta_A=\pi\delta_1+(1-\pi)\delta_0\).

Assume a continuous outcome with independent observations and normal errors of known variance \(\sigma^2\) in each treatment arm and biomarker stratum. Treatment is randomized within marker strata. The treatment effect is constant within each stratum, and stage 1 and stage 2 observations are independent. For the examples, use a one-sided test for benefit, with \(\alpha=0.025\). These assumptions make the sampling distributions and standard errors explicit; a real analysis must use an analysis method appropriate to its endpoint and design.

Stage 1 enrolls 80 participants per arm, allocated in fixed biomarker-stratum quotas: \(80\pi\) per arm with \(M=1\), and \(80(1-\pi)\) per arm with \(M=0\). The design selects the enriched route if the stage 1 positive-stratum statistic exceeds \(c=1.282\), the approximate 90th percentile of a standard normal distribution. Otherwise, it selects the all-comer route. Stage 2 is independent of stage 1. The enriched route enrolls 40 participants per arm in \(M=1\); the all-comer route enrolls 30 per arm in each marker stratum. Only the selected population is tested at the final analysis.

In the enriched route, the final estimator is the stage 2 treatment-minus-control difference among \(M=1\) participants. In the all-comer route, it is the prevalence-weighted difference, using the known design prevalence \(\pi\) and the two stratum-specific treatment contrasts. In either route, success means that the final one-sided \(Z\) statistic exceeds 1.96. The final test uses only independent stage 2 data; stage 1 determines the route, not the final test statistic.

What the Simulation Estimates

For each simulated trial, generate stage 1 outcomes under a chosen scenario \((\pi,\delta_1,\delta_0,\sigma)\), apply the selection rule, generate independent stage 2 outcomes for the selected route, and record the route, final result, and sample size. Repeat this complete process \(R\) times. The selection probability is the fraction routed to enrichment. The population-specific success rates are the fractions of all simulated trials that finish with a successful enriched-population claim or a successful all-comer claim, respectively. These rates include both route selection and conditional final-test success.

For fixed stratum sample sizes, the standard errors under the model are

SE_1=\sigma\sqrt{\frac{2}{n_{1+}}},\qquad SE_E=\sigma\sqrt{\frac{2}{40}},\qquad SE_A=\sigma\sqrt{2\left(\frac{\pi^2}{30}+\frac{(1-\pi)^2}{30}\right)}.

Here \(n_{1+}=80\pi\) is the stage 1 positive-stratum sample size per arm. The all-comer variance uses the squared prevalence weights because the stratum-specific contrasts are independently estimated. For a treatment effect \(\delta\), a one-sided \(Z\) test with standard error \(SE\) has power \(P(Z>1.96)\), where \(Z\) is normal with mean \(\delta/SE\) and variance 1 under the scenario.

Justification: Under the normal model, the stage 1 selection statistic has distribution \(N(\delta_1/SE_1,1)\). Thus the enrichment probability is \(1-\Phi(c-\delta_1/SE_1)\). Conditional on either route, independent stage 2 data give a valid level-\(\alpha\) test when that route’s treatment-effect null is true. Because exactly one route is tested, and selection uses no stage 2 outcomes, the chance of a false selected-population claim is at most \(\alpha\) when that selected population’s null is true. This conditional argument depends on the independent confirmation stage and the prespecified test; reusing stage 1 evidence in the final test would require a different valid procedure.

Worked Example: Type I Error Under the Global Null

Suppose \(\pi=0.25\), \(\delta_1=\delta_0=0\), and \(\sigma=1\). Stage 1 has 20 participants per arm in the positive stratum, so

SE_1=\sqrt{\frac{2}{20}}=0.3162,\qquad P(\text{select enrichment})=1-\Phi(1.282)=0.100.

Both possible final tests are level 0.025 under this global null. Since the final test is independent of selection, the overall probability of any false success is

$$ 0.100(0.025)+0.900(0.025)=0.025. $$

The population-specific success rates are 0.00250 for an enriched claim and 0.0225 for an all-comer claim; they sum to 0.025. In \(R=100{,}000\) runs, the expected counts are 10,000 enrichment selections and 2,500 total successes. These are expectations, not guaranteed simulation output. The Monte Carlo standard error for an estimated 0.025 success rate is approximately \(\sqrt{0.025(0.975)/100{,}000}=0.000494\).

Worked Operating Characteristics

Worked Example: A Larger Effect in the Positive Stratum

Use \(\pi=0.25\), \(\delta_1=0.60\), \(\delta_0=0\), and \(\sigma=1\). Stage 1 has \(n_{1+}=20\) per arm. The selection probability is

1-\Phi\left(1.282-\frac{0.60}{\sqrt{2/20}}\right) =1-\Phi(-0.615)=0.731.

For the stage 2 enriched test, \(SE_E=\sqrt{2/40}=0.2236\), so its conditional power is

1-\Phi\left(1.96-\frac{0.60}{0.2236}\right) =1-\Phi(-0.723)=0.765.

The all-comer effect is \(\delta_A=0.25(0.60)+0.75(0)=0.150\). Its standard error is \(\sqrt{2(0.25^2+0.75^2)/30}=0.2041\), and conditional power is \(1-\Phi(1.96-0.150/0.2041)=0.110\). Therefore, the success rates per trial are \(0.731(0.765)=0.559\) for an enriched claim and \(0.269(0.110)=0.0296\) for an all-comer claim. Overall success probability is approximately 0.589. This is not the conditional power of either route: it combines route selection and route-specific power.

Stage 1 enrolls 160 participants. Stage 2 enrolls 80 participants if enriched and 120 if all-comer. Expected total sample size is therefore

160+0.731(80)+0.269(120)=250.8\text{ participants}.

Worked Example: Changing Prevalence and Effects

Now let \(\pi=0.60\), \(\delta_1=0.40\), \(\delta_0=0.20\), and \(\sigma=1\). Stage 1 has \(n_{1+}=48\) per arm, giving \(SE_1=\sqrt{2/48}=0.2041\). Thus

P(\text{select enrichment}) =1-\Phi\left(1.282-\frac{0.40}{0.2041}\right) =1-\Phi(-0.678)=0.751.

The enriched stage 2 test has power \(1-\Phi(1.96-0.40/0.2236)=0.432\). The all-comer effect is \(\delta_A=0.60(0.40)+0.40(0.20)=0.320\). Its standard error is \(\sqrt{2(0.60^2+0.40^2)/30}=0.1862\), giving conditional power \(1-\Phi(1.96-0.320/0.1862)=0.405\). The respective success rates per trial are \(0.751(0.432)=0.324\) and \(0.249(0.405)=0.101\); total success probability is approximately 0.425. Expected sample size is \(160+0.751(80)+0.249(120)=250.0\) participants, to the nearest participant. A higher prevalence changes both the stage 1 precision for the positive subgroup and the all-comer effect and variance; it does not guarantee higher overall success.

Simulation Checks and Failure Modes

Before interpreting a large run, verify the implementation against the normal calculations above. Under the global null, the simulated selection fraction should be near 0.10 and the total success fraction near 0.025, allowing for Monte Carlo error. Under alternatives, compare simulated route-specific success rates with selection probability multiplied by conditional power. Large discrepancies usually point to a coding or design-definition error, not a surprising property of enrichment.

Mixing stage 1 and stage 2 evidence: If the final test reuses selection data but is analyzed as independent confirmation, its type I error may exceed the nominal level. Keep data roles separate or specify a procedure that accounts for adaptation.
Using the wrong all-comer effect: The all-comer contrast is prevalence-weighted, not the unweighted average of subgroup effects unless \(\pi=0.5\).
Ignoring the route in success rates: Conditional power describes a selected route. A per-trial population-specific success rate also multiplies by the probability of selecting that route.
Confusing Monte Carlo error with design uncertainty: Increasing \(R\) reduces simulation noise, but does not make assumed prevalence, effects, or variances more realistic.

An alternative is to calculate operating characteristics analytically from normal probabilities, as in the examples. That is efficient for this simple model and useful for validating code. Simulation is more flexible when recruitment, sample sizes, endpoints, or decision rules are complicated, but it requires careful implementation and sufficiently many replicates. Neither approach protects against incorrect assumptions. This simulation also treats prevalence and stratum quotas as known and fixed, assumes normally distributed outcomes with known variance, and does not model missing outcomes, imperfect marker classification, or recruitment delays.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

Under the global null, why does an independent final test keep the false-success probability at 0.025 even when the enrichment route is selected using stage 1 data?
If the enrichment selection probability is 0.60 and conditional enriched-route power is 0.70, what is the per-trial enriched-claim success rate?
For \(\pi=0.20\), \(\delta_1=0.50\), and \(\delta_0=0\), calculate the all-comer effect.
With selection probability 0.40, stage 1 size 160, enriched stage 2 size 80, and all-comer stage 2 size 120, calculate expected total sample size.

Solutions. (1) Conditional on the selected route, its independent stage 2 test has false-positive probability 0.025 under that route’s null; averaging over routes leaves that probability unchanged. (2) \(0.60(0.70)=0.42\), or 42%. (3) \(\delta_A=0.20(0.50)+0.80(0)=0.100\) outcome units. (4) \(160+0.40(80)+0.60(120)=264\) participants.