Simulation Diagnostics for Selection Bias Corrections

Why Simulate Selection Corrections?

After an interim analysis in a multi-arm multi-stage (MAMS) trial, the arms that continue are not a random subset of the original arms: continuation depends on observed data. An estimator that performs adequately before selection can therefore behave differently for a selected arm. Repeated-trial simulation provides a direct way to assess that behavior under specified treatment effects, trial conduct, and selection rules.

The aim is not to prove that a correction works for every possible trial. It is to measure how competing estimators behave across repeated trials generated from a stated scenario, and to identify conditions under which their bias, precision, or interval coverage deteriorates. The target of evaluation must be defined before calculating any performance measure.

Definition: In a simulated MAMS trial, let \(\theta_j\) be the true treatment effect for arm \(j\), and let \(J\) be the arm selected by the prespecified interim and final selection rules. For a method \(m\), let \(\hat{\theta}_{m,J}\) be its estimate for the selected arm. For evaluating the effect of the arm actually selected in each trial, the target is \(\theta_J\), and the estimation error is \(e_m=\hat{\theta}_{m,J}-\theta_J\). Bias, root mean squared error (RMSE), and coverage are calculated from these errors and the method’s reported intervals across simulated trials.

Specify the Simulation Before Comparing Methods

A useful simulation replicates the whole trial process, not just the final calculation. Specify the number of arms, allocation, stage sizes, outcome distribution, treatment effects, interim information, selection rule, and method used to construct each estimate and interval. Include relevant shared-control dependence, missingness, or other design features when they are part of the scenario. The scenario should state whether the model used by an estimator agrees with the data-generating model.

For each replication, generate the stage data, apply the selection rule exactly as it would be applied in the trial, and then calculate every candidate estimator and interval for the selected arm. If an arm is dropped, it should not be treated as selected merely because its simulated effect happens to be large. A method that cannot produce a result in some simulated trials needs a prespecified handling rule; silently discarding those trials can distort the performance summaries.

With \(R\) independent replications under the same scenario, define the simulation estimates of bias and RMSE as follows. The RMSE is measured in the same units as the treatment effect, while its square is the mean squared error.

\widehat{\operatorname{Bias}}_m=\frac{1}{R}\sum_{r=1}^{R}e_{m,r}, \qquad \widehat{\operatorname{RMSE}}_m= \sqrt{\frac{1}{R}\sum_{r=1}^{R}e_{m,r}^{\,2}}.

If method \(m\) reports an interval \([L_{m,r},U_{m,r}]\), its empirical coverage for the selected-arm target is the fraction of replications in which that interval contains the true effect of the selected arm:

\widehat{C}_m=\frac{1}{R}\sum_{r=1}^{R} I\{L_{m,r}\leq\theta_{J_r}\leq U_{m,r}\}.

Here \(I\{\cdot\}\) is 1 when the condition holds and 0 otherwise. For an interval procedure intended to have 95% coverage, a simulation estimate near 0.95 is desirable, but it is subject to Monte Carlo variation. For independent replications, the estimated coverage has approximate Monte Carlo standard error \(SE_{\mathrm{MC}}(\widehat{C}_m)=\sqrt{\widehat{C}_m(1-\widehat{C}_m)/R}\). This quantifies simulation noise; it is not the standard error of a treatment-effect estimate within one trial.

Interpret the Measures Together

Bias describes average signed error: positive bias means estimates tend to exceed their targets, and negative bias means they tend to fall below them. RMSE describes the typical error magnitude while penalizing large errors; it combines both systematic bias and variability. In particular,

\widehat{\operatorname{RMSE}}_m^{\,2} = \widehat{\operatorname{Bias}}_m^{\,2} + \frac{1}{R}\sum_{r=1}^{R} (e_{m,r}-\widehat{\operatorname{Bias}}_m)^2.

This follows by writing \(e_{m,r}=(e_{m,r}-\widehat{\operatorname{Bias}}_m)+\widehat{\operatorname{Bias}}_m\), squaring, and averaging. The cross-product averages to zero because the centered errors sum to zero. Thus, low bias alone does not guarantee low RMSE: a method with errors that vary widely can have a large RMSE even when its average error is close to zero.

Coverage evaluates the interval procedure, not just the point estimate. A procedure can have good average coverage while missing badly for a particular selection pathway, or while producing unnecessarily wide intervals. Record interval width as a complementary measure, and examine coverage by relevant scenario features. Do not interpret nominal 95% coverage as a guarantee that an interval from one observed trial contains the true effect.

Worked Example: Calculate Bias and RMSE From Replicated Errors

Suppose ten hypothetical simulated trials select an arm whose treatment effect is measured in points. The errors \(\hat{\theta}_{m,J}-\theta_J\) are recorded for two methods. Method A’s errors are \(2,1,0,3,-1,2,1,0,2,0\); Method B’s errors are \(-1,0,1,1,-2,0,1,-1,0,1\). These values are for illustration, not results from a real trial.

For Method A, the errors sum to 10, so its estimated bias is \(10/10=1.00\) point. Its squared errors sum to \(24\), giving

\widehat{\operatorname{RMSE}}_A =\sqrt{\frac{24}{10}} =\sqrt{2.4} =1.549\text{ points}.

For Method B, the errors sum to 0, so its estimated bias is \(0/10=0\) points. Its squared errors sum to \(10\), giving

\widehat{\operatorname{RMSE}}_B =\sqrt{\frac{10}{10}} =1.000\text{ point}.

In this small illustration, Method B has both less estimated bias and lower RMSE. Ten replications are far too few for a stable assessment of performance, however. In particular, zero estimated bias in ten trials does not show that the method has zero bias under repeated use.

Selection Changes Which Performance Questions Matter

The selected-arm target \(\theta_J\) varies across replications when different arms can be selected. This defines an unconditional performance measure over the trial’s selection pathways. It answers: across trials generated under this scenario, how well does the method estimate the true effect of whichever arm the rule selects?

A complementary analysis conditions on a selection event. For example, estimate bias or coverage among trials in which a particular arm is selected, or among trials in which selection occurs at an interim boundary. This answers a different question: how well does the method behave within that subset of trials? State the event and its frequency, because a conditional summary based on few replications may be highly variable. Do not confuse this random selected-arm target with a fixed target such as the effect of a prespecified arm; those are different estimands and should be assessed separately if both matter.

Worked Example: Overall Coverage Can Hide a Boundary Problem

In a hypothetical simulation of 1,000 trials, a selection-aware interval covers the true effect in 941 trials. Of those trials, 200 involved selection at a specified interim boundary; the interval covered in 164 of them. The other 800 trials had 777 covered intervals.

Overall empirical coverage is \(941/1000=0.941\), or 94.1%. Its Monte Carlo standard error is approximately

SE_{\mathrm{MC}}(\widehat{C}) =\sqrt{\frac{0.941(1-0.941)}{1000}} =0.00745.

For boundary-selected trials, coverage is \(164/200=0.820\), or 82.0%. In the other trials, it is \(777/800=0.97125\), or 97.1%. The counts reconcile: \(164+777=941\). The overall result is therefore not an adequate description of performance across selection pathways; it averages poor boundary coverage with much higher coverage elsewhere.

The boundary subgroup’s Monte Carlo standard error is approximately \(\sqrt{0.820(0.180)/200}=0.0272\). That uncertainty is substantial, so the apparent shortfall deserves follow-up with more replications and a review of the estimator and interval construction. The subgroup result is a diagnostic signal, not by itself an explanation of why undercoverage occurred.

Compare Methods Fairly and Check for Failure

When feasible, evaluate competing estimators on the same simulated trial replications. This means using the same generated outcomes and selection path for each method, then calculating each method’s estimate and interval. Such paired comparisons reduce irrelevant differences caused by one method receiving more favorable simulated data. They do not remove Monte Carlo uncertainty, and they do not make the methods’ errors independent.

Compare methods across a scenario grid rather than selecting a single convenient data-generating model. Include cases that vary treatment-effect patterns, outcome variability, allocation, selection thresholds, and agreement between the estimator’s outcome model and the generating model. Sensitivity to such assumptions is especially important when prior checks have identified plausible departures from the assumed model. A correction that performs well only when its assumptions hold should not be described as generally reliable.

Common diagnostics include plotting the distribution of errors, inspecting extreme errors, tabulating which arm was selected, and reporting coverage and interval width by selection event. Check that simulated allocation and selection frequencies agree with the specified design. Look for failures such as an estimator returning undefined values, intervals with zero or implausible width, or a few extreme errors dominating RMSE. Verify that every replication uses the same prespecified decision rules and that the true target recorded for evaluation is \(\theta_{J_r}\), not automatically the effect of the arm that would have been selected under a different rule.

Worked Example: Compare Methods Across Two Scenarios

Consider a hypothetical simulation with 5,000 paired replications per scenario. Scenario A generates outcomes under the estimator’s stated outcome model. Scenario B changes the outcome distribution while retaining the same MAMS selection rule; the correction method’s model is then misspecified. The following illustrative summaries use the selected-arm effect as the target. Coverage counts are out of 5,000.

Scenario	Method	Bias (points)	RMSE (points)	Covered intervals	Coverage
A	Naive	1.20	2.10	4,020	80.4%
A	Corrected	0.10	1.70	4,745	94.9%
B	Naive	1.05	2.05	4,080	81.6%
B	Corrected	0.55	1.90	4,420	88.4%

For example, Scenario B’s corrected-method coverage is \(4420/5000=0.884\), or 88.4%. The correction improves on the naive method in RMSE and coverage in both scenarios, but its estimated bias increases from 0.10 to 0.55 points and its coverage falls under misspecification. This pattern supports a qualified conclusion: the correction performs better than the naive method in these specified scenarios, but the results do not establish nominal coverage under the misspecified scenario.

Because replications are paired, a further comparison can calculate the difference in squared errors within each replication. That directly examines whether one method tends to reduce squared error on the same simulated trials. The table alone does not provide the variability of those paired differences, so it cannot establish how precisely the RMSE difference has been estimated.

Monte Carlo Precision, Alternatives, and Limitations

Choose the number of replications to make Monte Carlo error small enough for the decision being considered. For example, if estimated coverage is near 0.95 and \(R=1{,}000\), its Monte Carlo standard error is about \(\sqrt{0.95(0.05)/1000}=0.00689\), or 0.69 percentage points. Increasing \(R\) reduces this simulation error at a rate proportional to \(1/\sqrt{R}\). More replications do not correct a wrongly specified data-generating model.

Analytic calculations can be faster and can clarify behavior under idealized assumptions, but they may rely on approximations that do not capture the full adaptive selection process. Simulation can represent a complicated sequence of stages and selection decisions directly, but its conclusions remain conditional on the scenarios, implementation, and estimands chosen. Neither approach replaces the other: use analytic results where their assumptions are appropriate, and use simulation to examine performance under explicitly specified trial mechanisms and departures from assumptions.

Report the data-generating assumptions, selection rule, number of replications, methods, target, and treatment-effect units. Give bias, RMSE, coverage, and Monte Carlo precision where relevant, and show important scenario-specific results rather than only a favorable aggregate. A simulation assesses performance under its stated conditions; it cannot guarantee the same performance in a future trial whose conditions differ.

Common Pitfalls

Simulating only the selected-stage data. This omits the data-dependent pathway that created selection. Generate and apply the full trial process.
Using the wrong target. Comparing every estimate with one fixed arm effect does not evaluate the effect of whichever arm was selected. Define the target to match the scientific question.
Reporting bias without RMSE. Positive and negative errors can cancel. Also report error magnitude, such as RMSE, and inspect the error distribution.
Reporting only overall coverage. Good average coverage can conceal poor coverage after a particular selection event. Examine important pathways and report their replication counts.
Treating simulation output as exact. Performance estimates have Monte Carlo error. Increase replications when that error is too large for the comparison at hand.
Equating a successful simulation with a validated method. A method can perform well under one generating model and poorly under another. Assess plausible alternatives and state the scope of the evidence.

Key takeaway: A selection-correction simulation is informative only when its target, data-generating scenario, and complete selection process are explicit. Use bias, RMSE, and coverage together, quantify Monte Carlo uncertainty, and inspect performance across the selection pathways that matter.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In each simulated trial, a different arm may be selected. What target should be used to assess estimation of the selected arm’s effect?
A method has estimated bias near zero but large RMSE. Is that possible? Explain.
An interval covers in 920 of 1,000 independent simulated trials. Calculate its empirical coverage and approximate Monte Carlo standard error.
Why might an overall coverage estimate conceal a problem with intervals after boundary selection?
What is the advantage of applying two competing methods to the same simulated trial replications?

Solutions. (1) For replication \(r\), use \(\theta_{J_r}\), the true effect of the arm selected in that replication; condition-specific analyses should state the selection event. (2) Yes. Errors can average to nearly zero through cancellation while still varying widely, and RMSE includes both variability and squared bias. (3) Coverage is \(920/1000=0.920\), or 92.0%. Its approximate Monte Carlo standard error is \(\sqrt{0.920(0.080)/1000}=0.00858\), or 0.858 percentage points. (4) The aggregate combines trials from different selection pathways, so high coverage in common pathways can mask undercoverage in a subgroup. (5) Both methods face the same simulated data and selection path, making their comparison less affected by random differences between simulation samples; Monte Carlo uncertainty still remains.