Why Information Contribution Is Not an Enrollment Count
Two bridging studies can enroll the same number of participants yet provide quite different evidence about regional differences. One may allocate too few participants to the region of greatest interest, collect sparse exposure data, or leave too few safety events to estimate a regional contrast usefully. A larger study can also add little decision-relevant information if its participants contribute measurements that are already well represented.
The useful comparison is therefore not simply “which design enrolls more?” It is “under the assumptions and analysis planned, how often would each design estimate the regional differences well enough to support the intended decisions?” Simulation can answer that question by generating many hypothetical data sets under specified conditions and applying the planned analysis to each. The resulting operating characteristics describe the design’s expected performance, not the outcome of a real study.
Begin with the estimands and decisions. An exposure estimand might be the difference in mean log exposure between regions, with its exponent interpreted as a ratio of geometric means. A response estimand might be a regional difference in predicted treatment response at a specified dose or exposure. A safety estimand might be a difference in event risk over a stated follow-up period. These are not interchangeable: their units, models, precision, and decision thresholds differ.
The earlier material on targeted bridging study design motivates defining the population, estimands, and planned analysis before choosing the sample allocation. Here the emphasis is on comparing candidate designs by their expected performance for those estimands. A simulation is informative only when its data-generating assumptions, analysis method, and pass criteria correspond to the question being asked.
What to Simulate and What to Summarize
For each candidate design, specify how participants are allocated by region and treatment, which measurements are collected, the timing and amount of sampling, and the analysis that will be applied. Specify the assumptions used to generate exposure, response, and safety outcomes, including relevant variability and within-participant dependence. These assumptions need to be scientifically plausible and sufficiently explicit to reproduce the simulation.
For replicate \(b\), generate a hypothetical data set under the chosen design, fit the planned analysis, and record the estimates, standard errors, intervals, and decision outcomes. Repeat this process \(B\) times. For example, if \(I_b=1\) when replicate \(b\) meets a stated criterion and \(I_b=0\) otherwise, the estimated probability of meeting that criterion is:
This is a simulated operating characteristic. It is conditional on the scenarios and assumptions used to generate the data, and it has Monte Carlo error because it is estimated from a finite number of replicates. A design comparison should report the criteria and scenarios alongside the probabilities; a bare statement that one design “has more information” is not enough.
Criteria should match the decision. Examples include requiring an interval to be narrower than a prespecified precision target, requiring a confidence interval to exclude a clinically unacceptable response difference, or requiring a safety risk-difference upper bound to remain below an agreed limit. A joint criterion may require all of these conditions to hold in the same simulated replicate. That joint probability is not the same as any one endpoint’s pass probability.
Worked Example: A Regional Allocation and a Joint Criterion
Hypothetical example. Suppose a study compares mean log exposure between a local region and a reference region. Assume independent observations, a common standard deviation of \(0.35\) log units, and a true regional difference of \(m=0.2231\) log units. Candidate design A enrolls 40 local and 100 reference participants. For planning, use the normal approximation for the standard error of the difference:
A one-sided 95% lower confidence bound for an estimated difference \(\hat{\Delta}\) is \(\hat{\Delta}-1.645SE\). At the true difference, its expected value is:
Thus, under this planning calculation, the expected lower bound is above zero. This calculation is an endpoint-specific summary; it is not the probability that the whole study meets several criteria.
Now suppose 100,000 hypothetical data sets are simulated under the full planned exposure-response-safety model. A replicate passes only if the exposure criterion is met, the response interval satisfies its prespecified precision rule, and the safety upper bound satisfies its prespecified risk rule. If 83,540 replicates pass, then:
The result is a simulated joint pass probability of about 83.5% for this hypothetical design and scenario. It does not follow from the lower-bound calculation alone: it also reflects the response and safety criteria, their variability, and their dependence in the simulations. The distinction prevents a correct endpoint calculation from being misreported as a joint study-level probability.
Why Allocation and Measurement Matter
For two independent regional sample means with a common standard deviation \(\sigma\), the variance of their difference is \(\sigma^2(1/n_L+1/n_R)\), where \(n_L\) and \(n_R\) are the regional sample sizes. This follows because the variance of the difference of independent estimates is the sum of their variances, and each sample mean has variance \(\sigma^2/n\). Consequently, adding participants to a region that already has a much larger sample may contribute less precision than adding them to the smaller region.
Worked Example: Comparing Two Allocations at Equal Enrollment
Hypothetical example. Compare design A above, with 40 local and 100 reference participants, against design B, with 80 local and 60 reference participants. Both enroll 140 participants. Retain the assumptions of independent observations and common standard deviation \(0.35\) log units.
For design A, \(SE_A=0.35\sqrt{1/40+1/100}=0.06548\) log units. For design B:
The reduction is \((0.06548-0.05977)/0.06548\approx0.0872\), or about 8.72%. Using a two-sided 95% normal interval, the half-widths are \(1.96(0.06548)=0.1283\) and \(1.96(0.05977)=0.1171\) log units, respectively. Under this simple model, design B estimates the exposure difference more precisely despite the same total enrollment.
This does not establish that design B is better overall. It could have less useful information about safety, response, or a different regional contrast. Nor does the calculation apply unchanged if variances differ, observations are dependent, or the planned analysis uses a more complex model. It provides one transparent check on the simulation rather than a substitute for evaluating all objectives.
A Practical Simulation Comparison
A defensible comparison follows a consistent sequence. First state the estimands and criteria, including units, interval levels, and thresholds. Next specify candidate designs and the data-generating assumptions, then simulate and analyze replicates using the planned workflow. Finally compare the resulting operating characteristics, their Monte Carlo uncertainty, and the resources required.
Name the regional exposure, response, and safety quantities to be estimated, and state which decisions depend on them.
Set regional sample sizes, measurement schedules, follow-up, and analysis methods. State the model assumptions used to generate each outcome.
Generate data under each design, apply the planned analysis, and record estimates, intervals, failures, and pass indicators.
Summarize precision, bias, coverage, and pass probabilities, and assess Monte Carlo error before making a design choice.
The process depends on assumptions. The simulation must represent the intended regional populations and trial conduct well enough for the design question; the analysis must be the analysis that would actually be used; and criteria must be prespecified rather than selected after seeing which design performs best. If model fitting fails in some replicates, that failure is itself a performance outcome and should not be silently discarded.
Worked Example: Is Additional Local Enrollment Worthwhile?
Hypothetical example. A team compares two designs using the same complete joint criterion as in the first example. It simulates 100,000 replicates per design. Design A has 40 local and 100 reference participants, costs 1.00 arbitrary budget unit, and passes in 83,540 replicates. Design C has 60 local and 100 reference participants, costs 1.18 units, and passes in 89,600 replicates.
The estimated improvement is \(0.8960-0.8354=0.0606\), or 6.06 percentage points. The cost per percentage-point increase is \(0.18/6.06\approx0.0297\) budget units per percentage point. This is a useful marginal comparison only if the cost accounting and pass criteria are appropriate to the decision; it is not a universal value measure.
For a simulated probability \(\hat p\) based on \(B\) independent replicates, the Monte Carlo standard error is approximately \(\sqrt{\hat p(1-\hat p)/B}\). For design A it is \(\sqrt{0.8354(0.1646)/100{,}000}\approx0.00117\), or 0.117 percentage points. For design C it is \(\sqrt{0.8960(0.1040)/100{,}000}\approx0.000965\), or 0.0965 percentage points. The 6.06-point difference is much larger than this simulation noise. That supports a real difference between the simulated operating characteristics under the stated assumptions, but it does not establish that those assumptions describe every plausible setting.
Diagnostics, Alternatives, and Limits
Simulation results require diagnostics. Check that generated values have plausible ranges and regional distributions, that fitted models recover known parameters in simple test cases, and that nominal interval coverage is reasonable when the data-generating model matches the analysis. Inspect convergence and failed fits by region and replicate. Compare at least one result with a simpler analytical calculation, such as the standard-error calculation above. Increase the number of replicates if Monte Carlo error is material to the design decision.
A useful comparison table makes the decision transparent:
| Approach | Useful for | Main limitation |
|---|---|---|
| Enrollment totals | Describing recruitment scale and broad feasibility | Do not show allocation, measurement quality, or decision performance |
| Analytical precision calculation | Quick checks for simple estimands and models | May omit nonlinear models, multiple endpoints, fitting failures, and joint criteria |
| Simulation-based operating characteristics | Comparing realistic candidate designs under a planned analysis | Depend on specified assumptions and can incur substantial computational and Monte Carlo uncertainty |
A common mistake is to reason, “Design A enrolls more participants, so it must contribute more information.” Enrollment alone does not determine precision or decision performance: allocation, endpoint variability, sampling schedule, regional representation, and the analysis all matter. The correction is to compare designs on explicitly defined estimands and operating characteristics, while reporting the assumptions that generated those results.
A second pitfall is to treat one favorable endpoint result as evidence that the whole bridge is adequate. An exposure contrast can be precise while a response estimate is uncertain or safety information remains sparse. Conversely, a joint pass probability can conceal which endpoint is limiting. Report both endpoint-specific summaries and any joint criteria.
Finally, simulated performance is conditional, not a guarantee. If the assumed data-generating model is misspecified, the simulated probabilities may not describe actual performance. This tutorial’s comparison concerns the information contributed by designs under stated scenarios; it does not resolve uncertainty about unmeasured effect modifiers. Conclusions should therefore be phrased as conditional design comparisons, with limitations and key assumptions made explicit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why can two studies with the same total enrollment have different precision for a regional contrast?
- For independent regional means with common standard deviation \(0.30\), compare the standard error for \(n_L=30,n_R=90\) with that for \(n_L=60,n_R=60\).
- In 20,000 simulated replicates, 17,000 meet a prespecified joint criterion. What is the estimated joint pass probability?
- Why should failed model fits be included in a simulation assessment rather than removed without comment?
Solutions. (1) Regional allocation, outcome variability, measurement schedules, and the planned analysis affect precision; the total alone does not. (2) For 30 and 90 participants, \(SE=0.30\sqrt{1/30+1/90}=0.06325\). For 60 and 60, \(SE=0.30\sqrt{1/60+1/60}=0.05477\). Under the stated independent, common-variance assumptions, the balanced allocation is more precise for this contrast. (3) The estimate is \(17{,}000/20{,}000=0.850\), or 85.0%. (4) Fit failures affect whether the planned analysis can produce usable estimates; excluding them can overstate the design’s operating performance.