Why Simulate Conflict Scenarios?
A borrowing model can produce a narrow posterior interval when historical and current trial data agree. That apparent precision is useful only if the borrowing remains defensible when the trials differ. Conflict-scenario simulation asks a direct operating-characteristic question: across repeated current trials generated under specified conditions, how biased are the posterior estimates, how often do credible intervals cover the current-trial parameter, and how much historical information is borrowed?
The purpose is not to find one universally best prior. It is to expose trade-offs under conditions chosen before examining the results. A model that performs well when historical information is relevant may perform poorly when the historical estimate is displaced from the current-trial truth. Conversely, a model that avoids that failure may give wider intervals when the evidence is concordant.
A scenario is defined by the true current-trial parameter and the relationship between that truth and the historical information. Concordance means the historical estimate is centered near the current truth; moderate and severe conflict represent increasingly large discrepancies. These labels must be tied to explicit parameter values or clinically meaningful differences, not chosen after seeing which model looks favorable.
A Normal-Model Setup
For a transparent comparison, suppose the current trial estimates a continuous treatment effect \(\theta\), measured in outcome units, and its sample mean satisfies \(\bar{x}\mid\theta\sim N(\theta,1/n)\). Assume observations are independent and normally distributed with known variance 1, and let \(n=25\). Thus the current-trial standard error is \(1/\sqrt{25}=0.2\). Treat the historical estimate \(m_H\) as fixed for this conditional illustration. The current truth is \(\theta=0\), and scenarios set \(m_H=\theta+d\), where \(d\) is the historical displacement in outcome units.
Compare three prespecified analyses. Naive pooling uses the prior \(\theta\sim N(m_H,0.04)\), which has precision 25 and acts like an additional 25 observations under the same known-variance model. A robustly discounted analysis uses \(\theta\sim N(m_H,1)\), with precision 1. The no-borrowing analysis uses a flat prior for \(\theta\). Both normal priors are proper; the flat-prior posterior is proper because the current normal likelihood is proper.
For either normal prior \(N(m_H,1/q)\), where \(q\) is its precision, conjugate updating gives posterior variance \(V=1/(n+q)\) and posterior mean \(\hat{\theta}=(n\bar{x}+qm_H)/(n+q)\). Define the historical weight \(a=q/(n+q)\). Then \(\hat{\theta}=(1-a)\bar{x}+am_H\): \(a\) is the fraction of the posterior mean’s weight assigned to the historical center in this model. For no borrowing, \(a=0\), the posterior mean is \(\bar{x}\), and the posterior variance is \(1/n\).
Deriving the Coverage Calculation
Suppose the fitted model reports a central 95% posterior credible interval. In this normal setup its endpoints are \(\hat{\theta}\pm1.96\sqrt{V}\). Under repeated sampling of the current trial, \(\hat{\theta}\) is normally distributed with mean \(\theta+ad\) and standard deviation \(s_{\hat{\theta}}=(1-a)/\sqrt{n}\). Therefore the repeated-sampling coverage probability, conditional on the fixed historical center, is
where \(\Phi\) is the standard normal cumulative distribution function. This follows by rewriting interval coverage as \(-1.96\sqrt V\leq\hat{\theta}-\theta\leq1.96\sqrt V\), then standardizing the normally distributed quantity \(\hat{\theta}-\theta\), whose mean is \(ad\). It is not correct to infer coverage from posterior interval width alone: the interval’s center can be systematically displaced from the true current effect.
Worked Example: Calculating Bias and Coverage Under Moderate Conflict
Let \(d=0.5\) outcome units, so \(m_H=0.5\) while the current-trial truth is \(\theta=0\). For the robustly discounted prior, \(q=1\), giving \(a=1/(25+1)=1/26\) and \(V=1/26\).
The posterior mean’s bias is \(ad=(1/26)(0.5)=0.01923\) outcome units. Its repeated-sampling standard deviation is \((25/26)(0.2)=0.1923\) outcome units. The posterior interval has half-width \(1.96\sqrt{1/26}=0.3844\) outcome units, so its full width is \(0.7688\).
Substitution into the coverage formula gives \( \Phi((0.3844-0.01923)/0.1923)-\Phi((-0.3844-0.01923)/0.1923) =\Phi(1.899)-\Phi(-2.099)\approx0.953 \). Thus the interval covers the current truth in about 95.3% of repeated current trials under this conditional scenario. The result is a repeated-sampling coverage statement, not a claim that the parameter has a 95.3% probability of being inside a particular interval.
Building and Running the Simulation
The following scenarios use the same current-trial sample size and outcome variance. The historical estimate is held fixed in each scenario, so the simulation evaluates performance conditional on the stated historical result. The concordant case has \(d=0\); the moderate-conflict case has \(d=0.5\); and the severe-conflict case has \(d=2\), all in outcome units. These values are hypothetical and are intended to demonstrate the design, not to represent a clinical study.
Set \(\theta=0\), \(n=25\), variance 1, and the scenario-specific \(m_H=\theta+d\). Generate each current-trial mean as \(\bar{x}^{(r)}=\theta+Z^{(r)}/5\), where \(Z^{(r)}\sim N(0,1)\).
For each generated \(\bar{x}^{(r)}\), calculate the posterior mean and central 95% interval under naive pooling, the discounted prior, and no borrowing. Using the same replicate for all methods makes their comparison less noisy.
Across \(R\) replicates, calculate mean posterior estimate minus \(\theta\) as empirical bias, the fraction of intervals containing \(\theta\) as empirical coverage, and the mean interval width. Record the model’s historical weight alongside these measures.
Worked Example: Interpreting Scenario-Simulation Results
For \(R=100{,}000\) replicates, a simulation of this setup should produce results close to the values below. The bias values and interval widths follow directly from the specified normal models; the coverage column gives representative rounded simulation results. Monte Carlo variation means an actual run need not match every last digit.
| Scenario | Analysis | Bias (outcome units) | 95% coverage | Interval width (outcome units) | Historical weight |
|---|---|---|---|---|---|
| Concordant, \(d=0\) | Naive pooling | 0.000 | 0.994 | 0.554 | 0.500 |
| Concordant, \(d=0\) | Discounted prior | 0.000 | 0.955 | 0.769 | 0.0385 |
| Concordant, \(d=0\) | No borrowing | 0.000 | 0.950 | 0.784 | 0.000 |
| Moderate, \(d=0.5\) | Naive pooling | 0.250 | 0.607 | 0.554 | 0.500 |
| Moderate, \(d=0.5\) | Discounted prior | 0.0192 | 0.953 | 0.769 | 0.0385 |
| Moderate, \(d=0.5\) | No borrowing | 0.000 | 0.950 | 0.784 | 0.000 |
| Severe, \(d=2\) | Naive pooling | 1.000 | 0.000 | 0.554 | 0.500 |
| Severe, \(d=2\) | Discounted prior | 0.0769 | 0.937 | 0.769 | 0.0385 |
| Severe, \(d=2\) | No borrowing | 0.000 | 0.950 | 0.784 | 0.000 |
The naive-pooling calculations can be checked explicitly. Here \(q=25\), \(a=25/50=0.5\), \(V=1/50\), and \(s_{\hat{\theta}}=0.1\). Its interval half-width is \(1.96\sqrt{1/50}=0.2772\). Under moderate conflict, the bias is \(0.5(0.5)=0.25\), and coverage is \(\Phi((0.2772-0.25)/0.1)-\Phi((-0.2772-0.25)/0.1)\approx0.607\). Under severe conflict, the bias is \(0.5(2)=1\); the same formula gives coverage of approximately \(2.4\times10^{-13}\), effectively zero. The rounded simulation result of 0.000 is therefore expected, not a positive coverage rate of 1.9%.
The discounted prior borrows much less, so its bias remains smaller as conflict grows. Its coverage nevertheless declines to about 0.937 in the severe scenario; robust discounting is not a guarantee of nominal coverage. The no-borrowing analysis has 0.950 coverage in all three scenarios here because its estimator is unbiased and its known-variance interval has the matching normal width. Its width is larger than naive pooling’s, including when the historical information is concordant.
Diagnostics, Comparisons, and Limitations
For a coverage proportion \(\hat{C}\) estimated from \(R\) independent replicates, its Monte Carlo standard error is approximately \(\sqrt{\hat{C}(1-\hat{C})/R}\). At coverage near 0.95 with \(R=100{,}000\), this is about \(0.00069\). At coverage near 0.607 it is about \(0.00154\). Monte Carlo error quantifies simulation noise, not uncertainty about whether the chosen scenarios represent future trials. When estimates are unstable, increase \(R\), especially for tail events, and report the simulation size and precision.
Interpret the measures together. Bias diagnoses systematic displacement; coverage checks whether uncertainty statements perform across repeated trials; interval width measures precision; and the historical weight describes borrowing under the specified model. A narrow interval is not evidence of good performance if coverage collapses. Nor does nominal coverage alone establish clinical usefulness if intervals are too wide for the decision at hand.
| Approach | Strength in this comparison | Important limitation |
|---|---|---|
| Naive pooling | High precision when the historical and current information are genuinely compatible. | Large historical weight can produce substantial bias and severe undercoverage when conflict is present. |
| Robustly discounted prior | Limits the historical weight in advance, reducing the conflict-induced bias in these scenarios. | A fixed discount does not automatically adapt to conflict, and its performance depends on the chosen prior variance. |
| No borrowing | Provides a useful current-data benchmark and avoids bias from the fixed historical center in this setup. | It forgoes historical information even in the concordant scenario and may be less precise. |
This illustration assumes a known variance, a single continuous parameter, a fixed historical estimate, and no nuisance-parameter or model-selection uncertainty. Real trials may require simulating both historical and current data, treatment allocation, missingness, covariates, and the planned decision rule. If historical sampling variability is relevant, generate historical data anew in each replicate rather than treating \(m_H\) as fixed. Also assess a range of clinically plausible displacements: a few hand-picked scenarios cannot establish robustness over every possible discrepancy.
A common mistake is to call a model robust because it uses a protective prior or because its posterior intervals are wider. That reasoning confuses a model feature with demonstrated operating performance. The correction is to prespecify plausible scenarios, simulate the full analysis, and examine bias, coverage, width, and borrowing together. In particular, do not calculate coverage from the posterior variance alone; the estimator’s repeated-sampling bias and variability also enter the coverage probability.
Worked Example: Checking Monte Carlo Precision
Suppose a simulation estimates 95% coverage from \(R=10{,}000\) replicates. The approximate Monte Carlo standard error is \(\sqrt{0.95(0.05)/10{,}000}=\sqrt{0.00000475}=0.00218\). Thus simulation estimates will commonly vary by a few thousandths around the underlying coverage. Increasing to \(R=100{,}000\) reduces the standard error to \(\sqrt{0.0475/100{,}000}=0.000689\). This reduction improves numerical precision, but it does not repair an unrealistic scenario, a misspecified analysis model, or an incorrectly coded interval.
Check Your Understanding
Answer each question, then compare with the solutions below.
- With \(n=25\) and a prior precision \(q=1\), what is the historical weight \(a\), and what does it describe?
- For the discounted prior in the worked moderate-conflict scenario, state the posterior interval width and the repeated-sampling bias.
- Why can naive pooling have a narrow interval but poor coverage under severe conflict?
- If estimated coverage is 0.90 from 10,000 replicates, calculate its approximate Monte Carlo standard error.
- What changes if the simulation is intended to average over historical sampling variability rather than condition on one historical estimate?
Solutions. (1) \(a=q/(n+q)=1/26=0.0385\). It is the historical center’s weight in the posterior mean under this normal model. (2) The width is \(2(1.96)\sqrt{1/26}=0.7688\) outcome units, and the bias is \(ad=(1/26)(0.5)=0.01923\) outcome units. (3) The interval is narrow because pooling gives the historical data substantial precision and reduces posterior variance; under severe conflict the posterior mean is displaced from the current truth, so the narrow interval misses that truth in nearly every repeated trial. (4) The approximate standard error is \(\sqrt{0.90(0.10)/10{,}000}=0.00300\). (5) Generate historical data in each replicate, derive its estimate, and then fit the prespecified analysis alongside the generated current data; summarize operating characteristics over both sources of sampling variation.