From Poor Performance to Component Attribution
Finding that a design performs poorly in a simulated scenario is a diagnosis, not yet an explanation. A high probability of failing a design target could reflect the adaptation rule, the information available at an interim analysis, an operational feature, or a combination of these. To learn which features contribute to a specified failure outcome, compare simulation runs in which a feature is deliberately changed while the other relevant conditions are held fixed.
The comparison is only as clear as its target. Before running it, define the failure event precisely: for example, failure to control the type I error rate above a specified limit, or failure to achieve a required probability of selecting an effective treatment. Do not combine several distinct failures into one indicator unless that combined outcome is itself the stated target. A component may increase one type of failure while decreasing another.
Let \(F_i(0)\) be the failure indicator for simulation replicate \(i\) under a reference configuration, and \(F_i(1)\) the indicator under a configuration with the component changed. Each indicator is 1 if the specified failure occurs and 0 otherwise. Write \(p_0=P(F_i(0)=1)\) and \(p_1=P(F_i(1)=1)\) for the corresponding failure probabilities under the simulation setup. The risk difference is \(\Delta=p_1-p_0\): a negative value means the modified configuration has lower simulated failure risk.
Designing a Controlled Simulation Comparison
State the scenario, failure rule, component change, and all inputs that must remain fixed. For example, if testing an interim decision rule, hold the data-generating assumptions, accrual pattern, analysis method, stopping boundaries other than the rule under study, and failure definition constant. If a comparison changes several features at once, it estimates the effect of that package of changes; it cannot isolate one feature.
When feasible, run both configurations on the same simulated underlying trial conditions. These paired runs can use the same generated patient outcomes, recruitment times, and other random inputs, with each design configuration applied to them. This reduces noise from differences between simulated trial populations. Pairing does not mean that the two design outcomes are identical, nor does it remove uncertainty; it makes the comparison depend on within-replicate differences.
With \(M\) paired replicates, estimate the failure risks by \(\hat p_0=\sum_i F_i(0)/M\) and \(\hat p_1=\sum_i F_i(1)/M\), and estimate the contrast by \(\hat\Delta=\hat p_1-\hat p_0\). The four paired outcomes can be summarized as both pass, both fail, reference fails and modified passes, or reference passes and modified fails. Only the last two categories determine the paired risk difference.
Worked Example: Testing a Decision-Rule Change
A hypothetical design-assurance simulation tests whether a modified interim decision rule reduces the probability of failing a pre-specified efficacy-selection target. In \(M=2{,}000\) paired simulated trials, the reference design fails 18 times and the modified design fails 10 times. Among the pairs, the reference fails while the modified design passes in 12 replicates; the reference passes while the modified design fails in 4.
The estimated risks are \(\hat p_0=18/2{,}000=0.009\), or 0.9%, and \(\hat p_1=10/2{,}000=0.005\), or 0.5%. Thus
The estimated reduction is 0.4 percentage points. The paired differences \(D_i=F_i(1)-F_i(0)\) are nonzero for 16 pairs: 12 are \(-1\), and 4 are \(+1\). Therefore \(\sum_i D_i^2=16\), and the estimated standard error of the paired mean difference is
A rough 95% Monte Carlo interval is \(-0.004\pm1.96(0.00200)\), or approximately \([-0.00792,-0.00008]\). Under this simulation setup, the modified rule has lower estimated failure risk; the interval suggests that Monte Carlo error alone is unlikely to explain the entire difference. The interval does not cover uncertainty about whether the simulated assumptions represent the trial setting.
Why Pairing Helps: A Short Justification
For each paired replicate, define \(D_i=F_i(1)-F_i(0)\). These values are \(-1\), 0, or 1, and their expected value is \(p_1-p_0=\Delta\). The sample mean of the \(D_i\) values is exactly \(\hat\Delta\). If the simulated pairs are independent and generated under the stated setup, the Monte Carlo variance of this mean is estimated by the sample variance of \(D_i\), divided by \(M\).
Since \(D_i^2=1\) for discordant pairs and 0 otherwise, the sample variance can be written as \((\sum_i D_i^2-M\hat\Delta^2)/(M-1)\). Dividing by \(M\) gives the standard-error formula used above. This calculation shows why the number of discordant pairs matters: if both configurations usually agree within a replicate, their difference can be estimated more precisely than it could from two unrelated sets of simulated trials with the same marginal failure rates.
The usual approximate interval \(\hat\Delta\pm1.96SE(\hat\Delta)\) is a large-sample Monte Carlo interval. It relies on independent simulation replicates and a sufficiently regular sampling distribution for the estimated contrast. With very few failures or discordant pairs, the approximation can be poor; report the counts and consider more simulation runs or an appropriate interval method. Do not interpret the interval as capturing uncertainty in unknown clinical parameters or in the choice of scenarios.
Interactions: Components May Work Together
A one-at-a-time comparison answers a local question: what changes when this component is altered at the particular settings of the other components? It does not show whether the component has the same contribution under other configurations. To examine two components, simulate all four combinations while keeping the rest of the setup fixed. An interaction is present when the effect of changing one component depends on the setting of the other.
Let \(p_{ab}\) be the failure probability when component A is set to \(a\) and component B to \(b\), where 0 denotes the reference setting and 1 the modified setting. The difference-in-differences contrast is \((p_{11}-p_{01})-(p_{10}-p_{00})\). A value near zero indicates similar risk differences for A at the two settings of B; a nonzero value indicates that the estimated effect of A changes with B. The contrast is scale-specific: interaction on the risk-difference scale need not match interaction on a ratio scale.
Worked Example: Two Design Components and Their Interaction
A hypothetical simulation evaluates component A (an interim information rule) and component B (a confirmation requirement). Each of the four configurations is run for 10,000 independent replicates under the same scenario. The counts of failures for the specified target are 2,000 when neither is modified, 1,400 with A only, 1,700 with B only, and 800 with both. The estimated risks are therefore \(p_{00}=0.20\), \(p_{10}=0.14\), \(p_{01}=0.17\), and \(p_{11}=0.08\).
The estimated effect of A when B is at its reference setting is \(p_{10}-p_{00}=0.14-0.20=-0.06\), a 6 percentage-point reduction. When B is modified, the estimated effect of A is \(p_{11}-p_{01}=0.08-0.17=-0.09\), a 9 percentage-point reduction. Thus the estimated difference-in-differences is
The point estimate suggests that A's reduction in failure risk is 3 percentage points larger when B is modified. If the four sets of runs are independent, an approximate standard error is
The approximate 95% interval is \(-0.03\pm1.96(0.00704)\), or \([-0.0438,-0.0162]\). In this simulation, the interaction contrast is distinguishable from zero at the level of Monte Carlo error. It supports reporting the combined effects rather than describing either component as having one invariant contribution.
Scenario Dependence and Attribution Limits
A component comparison is conditional on the scenario and failure definition used. If a rule improves performance under one accrual pattern but worsens it under another, a single pooled contrast can conceal that difference. Estimate and report contrasts by scenario when the design question concerns robustness across conditions. If scenarios are weighted to represent an average operating environment, state the weights and explain that the resulting contrast is an average over that specified distribution, not a property that holds in every scenario.
Attribution also depends on what “component” means. Replacing an interim rule may change the information used, the timing of a decision, and downstream sample size. Those consequences are part of the effect of that rule as implemented. If the goal is to isolate one mechanism within a larger package, additional configurations are needed to separate those mechanisms. When components interact, there may be no single, context-free percentage of failure that can be assigned to each one.
In a simulation, the conditions are controlled by construction, so a contrast can support a claim about the consequences of changing the simulated design under the simulated assumptions. It does not prove that a feature caused an observed failure in a real trial. Nor does a large contrast show that the changed feature is the only problem: other weaknesses may coexist, and removing one may expose another.
Diagnostics, Alternatives, and Common Mistakes
Before interpreting a contrast, check that both configurations used the same failure definition, scenario inputs, analysis conventions, and relevant random-number structure. Confirm that implementation logs show the intended change and no accidental differences. Report the failure counts as well as the estimated risks and contrast. If results vary substantially across batches of simulation runs, across scenarios, or under small reasonable changes to the setup, that instability is itself important design-assurance evidence.
| Approach | Useful for | Main limitation |
|---|---|---|
| One-at-a-time controlled contrast | Estimating a component's effect at a fixed setting of other components | Can miss interactions and depends on the chosen reference configuration |
| Factorial comparison | Estimating component effects and interactions across combinations | Requires more configurations; estimates remain specific to the simulated scenario |
| Uncontrolled comparison of past runs | Identifying patterns that may motivate a controlled test | Differences may reflect multiple changed inputs, not the component of interest |
Common mistake: “The modified design failed less often, so the component we changed is the cause of poor performance.” This overstates the result. A controlled contrast estimates a change in simulated risk for a particular failure outcome, scenario, and implementation. It does not establish a unique cause, remove the need to assess Monte Carlo error, or guarantee the same effect in another scenario. A sound conclusion names the contrast, its uncertainty, and the conditions under which it was simulated.
For very rare failure events, a modest number of direct simulation runs may produce too few failures to estimate a component contrast precisely. In that case, report the limited information rather than treating zero observed failures as proof of zero risk. The basic attribution logic remains the same, but the estimation method must be appropriate to the rarity of the outcome.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- In 1,000 paired runs, the reference design fails in 30 and a modified design fails in 22. There are 15 pairs where only the reference fails and 7 where only the modified design fails. What is the estimated risk difference, and what does its sign mean?
- Why can an unpaired comparison of two configurations be less precise than a paired comparison using common simulated trial conditions?
- Suppose the risk difference for component A is \(-0.04\) with B at its reference setting and \(-0.01\) with B modified. What is the difference-in-differences, and what does it suggest?
- True or false: If a component change reduces simulated failure in one scenario, the simulation proves it caused poor performance in the real trial.
Solutions. (1) The difference is \(\hat\Delta=22/1{,}000-30/1{,}000=-0.008\), a reduction of 0.8 percentage points under the simulated setup. The discordant counts agree: \((7-15)/1{,}000=-0.008\). (2) Pairing controls variation in the simulated trial conditions, so the estimate uses within-pair differences rather than variation between unrelated sets of runs. (3) The contrast is \(-0.01-(-0.04)=0.03\). A's estimated reduction is 3 percentage points smaller when B is modified, indicating an interaction on the risk-difference scale. (4) False. The result supports a conditional claim about simulated risk under the specified assumptions; it does not prove a real-trial cause or establish that the effect holds in other scenarios.