Why the Assumed Model Is Not the Whole Calibration Story
A trial’s operating characteristics depend on what data are generated and on how the trial analyzes those data. Calibration that simulates only from the analysis model checks performance when the model is exactly right. It does not show what happens when outcomes are more variable, the analysis targets a different quantity than intended, or an assumption about observation fails. Those differences can change the probability of a favorable decision even when the decision rule itself is unchanged.
The central comparison is between an analysis model, which determines the likelihood and posterior used by the trial, and an alternative data-generating model, which represents a plausible way trial data might actually arise. The aim is not to declare every departure fatal. It is to measure how sensitive the trial’s false-positive rate, power, bias, and interval coverage are to specified departures.
For a Bayesian decision rule, for example, the analysis may declare success when the posterior probability that a treatment effect exceeds a clinically meaningful threshold is greater than a prespecified cutoff. The operating characteristic is evaluated by repeatedly generating trial data under a specified truth, fitting the same analysis model each time, and recording whether that rule declares success. A favorable posterior probability in one simulated trial is not itself an operating characteristic; the operating characteristic is the long-run frequency or expectation across trials.
Define the Comparison Before Simulating
A useful calibration specifies three items before calculation: the estimand (the quantity the trial is intended to learn about), the analysis model and decision rule, and the alternative data-generating model. Keep the decision rule fixed when comparing scenarios. Otherwise a change in performance could be due to a changed rule rather than to model misspecification.
Under the null, the false-positive rate is the probability that the rule declares success when the null condition for the target estimand holds. Under an alternative, power is the probability of success when that alternative is true. Bias is the expectation of the estimator minus the target estimand, and coverage is the probability that the reported interval contains that estimand. These quantities answer different questions: a decision rule can have an elevated false-positive rate even if the estimator is unbiased, and an estimator can be biased even if its interval is wide.
Here \(\theta\) is the estimand, \(\hat{\theta}\) its estimator, \(D\) the trial data, and \(C(D)\) the interval produced from those data. Probabilities and expectations are taken over repeated data sets under the specified data-generating model. For a Bayesian analysis, a credible interval has a posterior interpretation conditional on the observed data and analysis model. Its repeated-trial coverage is a separate property; it is not guaranteed by its posterior probability content.
The comparison must use the same estimand in both models. If an analysis coefficient corresponds to a conditional effect but the intended target is a marginal effect, a discrepancy between the coefficient and the target is an estimand mismatch as well as a possible model mismatch. Report the target explicitly rather than treating every fitted parameter as the same treatment effect.
Worked Example: Variance Misspecification Changes False Positives and Power
Worked Example: Variance Inflation Under the Null and Alternative
Suppose a trial uses a normal approximation for a treatment-effect estimate \(\hat{\theta}\). The analysis treats its standard error as \(s\), and the prior and likelihood yield an approximately normal posterior centered at \(\hat{\theta}\) with standard deviation \(s\). The rule declares success when the posterior probability of a positive effect exceeds 0.975. For this illustration, assume the posterior is normal and the prior is sufficiently weak that this rule is equivalent to \(\hat{\theta}/s>1.96\). Under the analysis model, the estimator’s sampling standard deviation is \(s\). Consider an alternative data-generating model under which it is \(\sqrt{1.90}s\), with the estimator still unbiased and normally distributed. The alternative changes variability, not the target or the decision rule.
Under the null, take \(\theta=0\). The standardized statistic \(Z=\hat{\theta}/s\) then has distribution \(N(0,1.90)\), so it can be written as \(\sqrt{1.90}U\), where \(U\sim N(0,1)\). The false-positive rate is therefore
The decision rule’s false-positive rate is about 7.752%, rather than the 2.5% upper-tail rate implied by the correctly specified analysis distribution. The source is clear: the analysis uses a threshold of 1.96 nominal standard errors, but the estimator varies more than the analysis assumes.
Now suppose the true effect is \(2.5s\). Under the alternative data-generating model, \(Z\sim N(2.5,1.90)\). The power is
With the assumed variance correctly specified, the corresponding power would be \(\Phi(2.5-1.96)=\Phi(0.54)\approx0.7054\). Under the alternative model, power falls to about 65.23%, while the false-positive rate rises. Both calculations use the same decision rule; only the sampling distribution changes.
The same variance mismatch also affects interval coverage. The analysis’s nominal 95% interval is \(\hat{\theta}\pm1.96s\). Under the null, it contains the true value when \(|Z|\leq1.96\). Thus its actual coverage is
So the interval covers the true effect in about 84.50% of repeated trials in this scenario, not 95%. The assumptions used here are explicit: the estimator is unbiased and normal under each scenario, the analysis uses standard error \(s\), and the decision threshold is 1.96. If the alternative distribution were skewed or heavy-tailed, these normal-tail calculations would not suffice; simulation from that distribution would be needed.
Worked Example: An Observation Assumption Can Bias the Estimate
Worked Example: Complete Cases Under Outcome-Dependent Observation
Consider a hypothetical randomized trial with a binary outcome. The intended estimand is the marginal risk difference, treatment risk minus control risk, for all randomized participants. The true event risks are 0.40 in control and 0.30 in treatment, so the target risk difference is \(0.30-0.40=-0.10\), or a 10-percentage-point reduction. Suppose the analysis uses only participants with observed outcomes and interprets their risk difference as the full randomized-population risk difference. Consider this alternative data-generating model: observation depends on the outcome, with the following probabilities.
In control, an event is observed with probability 0.50 and a non-event with probability 0.90. In treatment, the corresponding probabilities are 0.80 and 0.90. Randomization still makes the arms comparable at assignment, but outcome-dependent observation means the observed participants need not represent all randomized participants.
In control, the probability of an observed event is \(0.40(0.50)=0.20\), and the probability of an observed non-event is \(0.60(0.90)=0.54\). Therefore, among observed control outcomes, the event risk is
In treatment, the probability of an observed event is \(0.30(0.80)=0.24\), and the probability of an observed non-event is \(0.70(0.90)=0.63\). The observed treatment risk is
The complete-case risk difference is \(0.2759-0.2703=0.0056\), approximately a 0.56-percentage-point increase, rather than the true 10-percentage-point reduction. Its bias relative to the stated estimand is \(0.0056-(-0.10)=0.1056\), or 10.56 percentage points. This calculation is a population-level comparison, before adding sampling variability.
The key issue is not simply that some outcomes are missing. It is that the analysis treats selected observed outcomes as if they identified the full-population risks, although the specified observation probabilities depend on outcome. A different analysis may be appropriate under additional assumptions, but it cannot recover the intended estimand from the complete cases by asserting that randomization alone removes selection after randomization.
Worked Example: Bias and Extra Variability Both Reduce Coverage
Worked Example: Actual Coverage of a Nominal 95% Interval
Suppose an analysis reports a normal 95% interval \(\hat{\theta}\pm1.96s\) for a continuous treatment effect \(\theta\). Under the analysis model, \(\hat{\theta}\) is centered at \(\theta\) with standard deviation \(s\). To assess a departure, suppose the alternative data-generating model instead gives \(\hat{\theta}\sim N(\theta+0.5s,(1.2s)^2)\). This specified scenario has bias \(0.5s\) and sampling standard deviation \(1.2s\). The interval formula remains unchanged.
The interval covers \(\theta\) exactly when \(-1.96s\leq\hat{\theta}-\theta\leq1.96s\). Standardizing under the alternative model gives
The actual coverage is about 86.80%, below the nominal 95%. The interval is centered on an estimator whose repeated-trial distribution is shifted from the target and more variable than assumed. This example describes a particular sampling distribution; in an actual calibration, that distribution should be justified by the alternative data-generating model or estimated by repeated simulation.
A Practical Calibration and Diagnostic Sequence
A disciplined comparison makes the source of a performance change interpretable. In simulation, each trial replicate should be generated under one clearly stated scenario, analyzed with the same prespecified model, and evaluated against the same estimand and rule. For an estimated operating characteristic, the proportion of successful replicates estimates its probability; the earlier material on Monte Carlo uncertainty addresses the simulation error in that estimate.
Write down the estimand, null and alternative conditions, success criterion, and interval procedure. Specify whether success is based on a posterior probability, an interval, or another fixed decision rule.
Record the outcome distribution, dependence structure, covariates, missing-data assumptions, and prior that determine the fitted posterior or estimator. These assumptions define what the analysis treats as plausible.
Change a stated feature, such as outcome variability, observation mechanism, or treatment-effect structure, while keeping the estimand and decision rule clear. Do not label a scenario simply “misspecified”; say exactly what differs.
Apply the planned analysis to each simulated data set, then estimate false-positive rate, power, bias, and coverage as appropriate. Compare scenario results with the matched analysis-model scenario and explain which assumption accounts for the difference.
Diagnostics should address both the generated data and the fitted analysis. Check whether simulated event rates, variances, dependence, and observation patterns match the scenario specification. In fitted data, look for systematic residual patterns, unexplained dispersion, influential observations, unstable estimates, or posterior predictive discrepancies relevant to the assumed likelihood. Such checks can signal a mismatch, but they do not prove that the model is correct when no discrepancy is detected. In a simulation study, the data-generating mechanism is known by construction; this makes it possible to isolate the effects of the departures deliberately included.
A useful comparison is between calibration under the analysis model and calibration under specified alternatives. The first verifies behavior in the model’s own idealized setting. The second evaluates vulnerability to particular departures. A robust or differently specified analysis changes the procedure being evaluated; it can be useful to compare, but it should not be confused with measuring the planned analysis under misspecification. Similarly, a Bayesian posterior predictive check asks whether observed data are compatible with features of the fitted model; it is not, by itself, a repeated-trial guarantee of false-positive control or coverage.
Common Pitfalls and Limitations
- Calling a favorable posterior probability a calibrated error rate. A posterior probability is conditional on the observed data and model. False-positive rate and power are repeated-trial probabilities under specified truths.
- Changing the model and the rule at once. If both change, it is difficult to attribute an operating-characteristic difference. First hold the decision rule fixed to assess the impact of the alternative data generator.
- Reporting coverage without naming its target. Coverage must refer to a particular estimand. An interval for a conditional model parameter need not cover a marginal treatment effect.
- Assuming randomization repairs every analysis assumption. Randomization supports baseline comparability in expectation; it does not automatically correct outcome-dependent selection, an incorrectly computed standard error, or a mismatch between the fitted parameter and intended estimand.
- Interpreting a few scenarios as universal protection. Results apply to the data-generating models and parameter values actually examined. A small change in one scenario does not guarantee resilience to all forms of misspecification.
Misspecification calibration is therefore a structured stress test, not a certificate of robustness. Scenarios may simplify reality, and simulation cannot reveal departures that were neither anticipated nor represented. Its strength is that assumptions are made explicit and consequences are quantified on the same scale as the trial’s decisions.
Check Your Understanding
Answer each question using the stated model and estimand. Brief solutions follow the questions.
- If an unbiased estimator has actual standard deviation \(\sqrt{1.90}s\), while the analysis uses standard error \(s\) and a one-sided cutoff of 1.96, what is the false-positive rate under a normal null distribution?
- Why is a posterior probability computed in one simulated trial not the trial’s false-positive rate?
- In the missing-outcome example, what was the true marginal risk difference, and what was the complete-case risk difference?
- For an estimator distributed as \(N(\theta+0.5s,(1.2s)^2)\), what two departures from the analysis assumptions affect the nominal interval’s coverage?
- Name two items that should remain fixed when comparing a planned analysis with an alternative data-generating model.
Solutions. (1) \(1-\Phi(1.96/\sqrt{1.90})\approx0.07752\), or 7.752%. (2) It is conditional on that trial’s data and the assumed model; the false-positive rate is the repeated-trial probability of success when the null holds. (3) The true risk difference was \(-0.10\); the complete-case difference was approximately \(0.0056\). (4) The estimator is biased upward by \(0.5s\) and has standard deviation \(1.2s\), greater than the assumed \(s\). (5) Keep the estimand and decision rule fixed; doing so allows differences in operating characteristics to be attributed to the specified data-generating departure rather than a changed target or rule.