Why Simulate an Intermediate-Based Selection Rule?
An intermediate-based selection rule makes the later trial depend on data observed earlier. For each experimental arm, the rule may retain or drop that arm using an intermediate outcome, while the final objective is assessed on an event endpoint. The joint behaviour matters: participants who do well on the intermediate outcome may, or may not, tend to have better event outcomes. As a result, calculations that treat selection and final analysis as independent can misrepresent the design.
A simulation makes this dependence explicit. One simulated trial reproduces the planned randomization, generates both outcomes under a specified joint model, applies the selection rule, and performs the planned final analysis. Repeating that process estimates how often the design selects arms, detects treatment benefit, or makes a false claim. These estimates describe the design under the assumptions used to generate the data; they are not guarantees about a future trial.
A Joint Model for Intermediate and Event Outcomes
Consider a control group indexed by \(a=0\) and experimental arms indexed by \(a=1,\ldots,K\). For a concrete model, let \(R\) be a binary intermediate outcome, with \(R=1\) denoting response, and let \(T\) be time to the final event. In arm \(a\), specify the response probability \(p_a\), the event hazard \(\lambda_a\), and a dependence parameter \(\rho_a\). The following Gaussian-copula construction preserves the specified marginal distributions while inducing dependence between the outcomes.
The construction requires \(0<p_a<1\), \(\lambda_a>0\), and \(-1<\rho_a<1\). Conditional on treatment arm, assume participants’ outcome pairs are independent and identically distributed. These are modelling assumptions, not consequences of the simulation algorithm. If the design includes censoring, the simulation must also generate censoring times according to a stated model and the final analysis must use the observed event and censoring data. The examples below use complete event follow-up for clarity.
The dependence parameter is important because changing it can change the probability that an arm is both retained and successful at the final analysis, even when \(p_a\) and \(\lambda_a\) remain fixed. The copula is one useful model, not a uniquely correct description of biological dependence. Sensitivity analyses should therefore consider scientifically plausible dependence values.
Worked Example: Generate One Selection-Look Dataset
Suppose one experimental arm and a control arm each have four participants at the selection look. Use \(p_0=0.40\), \(p_1=0.60\), \(\lambda_0=0.20\) per month, and \(\lambda_1=0.14\) per month. Let both arms have \(\rho_a=0.50\). The following are hypothetical draws from the specified bivariate normal model.
For control, the four \((Z_I,Z_T)\) pairs are \((-0.50,-0.20)\), \((0.10,0.30)\), \((0.60,0.70)\), and \((1.00,0.80)\). The response threshold is \(\Phi^{-1}(1-0.40)=\Phi^{-1}(0.60)\approx0.253\). Thus, two of the four participants respond, giving \(\hat p_0=2/4=0.50\).
For the experimental arm, the pairs are \((-0.10,-0.30)\), \((0.20,0.50)\), \((0.50,0.10)\), and \((-0.70,-0.80)\). Its threshold is \(\Phi^{-1}(1-0.60)=\Phi^{-1}(0.40)\approx-0.253\). Three values of \(Z_I\) exceed this threshold, so \(\hat p_1=3/4=0.75\).
Suppose the prespecified rule retains the arm when \(\hat p_1-\hat p_0\geq0.10\). Here the difference is \(0.75-0.50=0.25\), so the arm is retained. As a check on the event-time transformation, the first experimental participant has \(Z_T=-0.30\), so \(\Phi(Z_T)\approx0.3821\) and \(T=-\log(1-0.3821)/0.14\approx3.44\) months. The value is positive, as required for a time, and the marginal event-time model uses the specified hazard of \(0.14\) per month.
From a Trial Rule to a Simulation Algorithm
The algorithm must mirror the protocol in its sequence and denominators. For example, if each arm and the control contribute \(n_1\) participants before selection, the intermediate comparison uses those \(n_1\) participants—not a later sample size. If an arm is retained, it and the control may then accrue \(n_2\) additional participants. A shared control group is generated once per stage and used in each retained-arm comparison; it must not be regenerated separately for each contrast.
Set the number of arms, stage sample sizes, \(p_a\), \(\lambda_a\), and \(\rho_a\), the selection criterion, and the final testing and multiplicity procedure. Define in advance which trial-level outcomes will be recorded.
Randomize participants to the planned arms and generate each participant’s intermediate and event outcomes jointly. Calculate the intermediate summaries and apply the exact arm-dropping rule.
For each retained arm, generate any planned additional participants. Combine the relevant data and apply the prespecified final event analysis and multiplicity procedure. A dropped arm cannot produce a final rejection under a rule that stops that arm.
Record which arms were retained and rejected, along with quantities such as sample size and events. Repeat independently for \(B\) simulated trials under the same scenario.
For a simple complete-follow-up illustration, compare a retained arm with the shared control using the logrank test. At each ordered event time \(j\), let \(Y_{1j}\) and \(Y_{0j}\) be the numbers at risk in the experimental and control groups, and let \(d_{1j}\) be the number of experimental-arm events. With continuous event times and no ties, the total number of events at that time is one. The logrank score and its variance under equal hazards are
A smaller hazard in the experimental arm tends to produce fewer experimental events than expected under equal hazards, so \(U_a\) tends to be negative. Define \(Z_{B,a}=-U_a/\sqrt{V_a}\); larger values favour benefit. For a one-sided test at level \(\alpha_a\), reject the equal-hazards null when \(Z_{B,a}\geq z_{1-\alpha_a}\), where \(z_{1-\alpha_a}\) is the corresponding standard-normal quantile. The simulation must use the design’s actual final test and error-control procedure; this particular complete-follow-up illustration is not a substitute for a different prespecified analysis.
Worked Examples of Operating Characteristics
Worked Example: Estimate Familywise Error Under the Global Null
Consider two experimental arms, each with a shared control. Under a hypothetical global-null scenario, both experimental arms have the same response probability and event hazard as control. Suppose the design uses a total one-sided familywise level of \(0.025\), with a Bonferroni final threshold of \(0.0125\) for each arm. Run \(B=10{,}000\) complete simulated trials, applying the intermediate-based selection rule before the final tests.
Suppose 231 replicates have at least one false final rejection. The estimated familywise error probability is \(\hat\theta=231/10{,}000=0.0231\), or 2.31%. Its Monte Carlo standard error is \(\sqrt{0.0231(1-0.0231)/10{,}000}\approx0.00150\), or 0.150 percentage points. The estimate is below 2.5%, but its Monte Carlo uncertainty is not negligible; one should not interpret this single estimate as proof that the true error probability is exactly controlled at 2.5%. The scenario, test assumptions, and implementation all matter.
Worked Example: Estimate Retention and Final Success
Now consider one experimental arm with \(p_0=0.40\), \(p_1=0.55\), \(\lambda_0=0.20\) per month, and \(\lambda_1=0.15\) per month. Let \(n_1=80\) per group at selection and \(n_2=120\) additional participants per group if the arm is retained. Retain the arm when the observed response-rate difference is at least \(0.05\), and test the final event endpoint one-sided at \(\alpha=0.025\). In \(5{,}000\) hypothetical simulated trials at \(\rho=0.50\), suppose the arm is retained in 3,600 trials and is both retained and finally rejected in 2,800.
The estimated retention probability is \(3{,}600/5{,}000=0.720\). The estimated probability of success under the complete design—selection and final rejection—is \(2{,}800/5{,}000=0.560\). Its Monte Carlo standard error is \(\sqrt{0.560(1-0.560)/5{,}000}\approx0.00702\). This joint-success estimate is not the same as the rejection proportion among retained trials: that conditional proportion is \(2{,}800/3{,}600\approx0.778\). The two answer different questions, so the operating characteristic must be defined before it is calculated.
Worked Example: Examine Sensitivity to Dependence
Use the design and marginal parameters in the previous example, but run 5,000 replicates at each of three latent correlations, changing only \(\rho\). Suppose hypothetical simulation results are as follows.
| \(\rho\) | Retained trials | Retained and rejected | Estimated joint success |
|---|---|---|---|
| 0.00 | 3,550 | 2,400 | 0.480 |
| 0.50 | 3,600 | 2,800 | 0.560 |
| 0.80 | 3,600 | 3,050 | 0.610 |
For \(\rho=0.80\), the estimated joint success is \(3{,}050/5{,}000=0.610\), with Monte Carlo standard error \(\sqrt{0.610(1-0.610)/5{,}000}\approx0.00690\). The retention estimates are similar across these runs, but the joint-success estimates differ. In this hypothetical model, stronger positive latent dependence can make selection on response more informative about event outcomes. These figures are illustrations, not empirical findings; other parameter values or a different dependence model could give a different pattern.
Diagnostics, Alternatives, and Limitations
Before interpreting operating characteristics, inspect whether the simulation is doing what the model and protocol specify. Useful checks include:
- Across generated participants, response proportions should be close to the specified \(p_a\), and event-time summaries should be consistent with the specified \(\lambda_a\).
- Selection decisions should agree with the stated rule when the intermediate summaries are checked directly, including equality at the cutoff.
- The recorded number of retained arms, participants, and events should respect the design’s stopping and accrual rules. A dropped arm should not contribute post-selection participants or a final rejection.
- Monte Carlo estimates should be reported with their simulation size and, especially for probabilities, a Monte Carlo standard error. Rare events may require substantially more replicates for useful precision.
- Dependence sensitivity should be examined over plausible values of \(\rho_a\), and, where appropriate, across alternative joint outcome models. A latent correlation parameter alone does not establish that the model captures the clinical relationship.
Simulation is especially useful when selection rules, shared controls, and dependent outcomes make analytic operating-characteristic calculations cumbersome. Analytic calculations or numerical integration can be faster and can clarify why a result holds under a simplified model. They are valuable cross-checks where available, but may require approximations or restrictive assumptions. Neither approach removes the need to specify the estimand, the selection rule, and the final test precisely.
A common mistake is to simulate intermediate and event outcomes independently because their marginal distributions are specified separately. That produces \(\rho_a=0\) in the latent construction, regardless of the intended clinical relationship, and can give misleading joint-success estimates. The correction is to specify and generate a joint model, then examine sensitivity to its assumptions. A second pitfall is counting final rejection only among retained trials and labelling it “power.” That is conditional power among selected trials; for the probability that the whole design succeeds, count a success only when the arm is selected and rejected, and divide by all simulated trials.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- In the Gaussian-copula construction, what marginal distribution does the event time have, and what does \(\rho_a\) represent?
- A simulated arm has 640 retentions in 1,000 trials and 420 trials with both retention and final rejection. Calculate the retention probability and the joint-success probability.
- Why should a shared control group be generated once rather than independently for each experimental-arm comparison?
- What is the difference between rejection probability among retained trials and the probability of success for the full selection design?
- If the observed response proportion is near its target but the event-time distribution is not, what should be checked before interpreting the simulated power?
Solutions. (1) The event time is exponential with hazard \(\lambda_a\); \(\rho_a\) is the correlation between latent standard-normal variables, not generally the observed response–time correlation. (2) Retention is \(640/1{,}000=0.640\); joint success is \(420/1{,}000=0.420\). (3) The same randomized control participants contribute to each contrast, so independent control samples would incorrectly remove shared-control dependence and alter the simulated design. (4) Conditional rejection divides by retained trials; full-design success divides the count of trials that are both retained and rejected by all trials. (5) Check the event-time transformation, hazard units and parameters, and the generated event-time marginal distribution; a mismatch means the simulation does not represent the stated model.)