Why Benchmark an External Control?
An external control can provide information about outcomes under a control regimen when the trial’s randomized control arm is small or unavailable. When a concurrent randomized control arm is available, it offers a useful benchmark: both estimates concern control outcomes, but they arise from different sources and may be affected by different biases. Comparing them can reveal discrepancies that need explanation before an external-control estimate is relied upon.
The comparison is meaningful only when the two estimates refer to the same outcome definition, follow-up period, control regimen, and target population. Differences in case mix matter as well. A higher crude event rate in the external cohort, for example, may reflect a larger proportion of high-risk patients rather than a source-related difference in outcomes. The earlier material on transport weights and posterior predictive checks is relevant background for these alignment and fit questions.
This discrepancy is a diagnostic comparison, not the treatment effect and not a direct measurement of bias against the unknown true control risk. Even a randomized control arm has sampling uncertainty, and an external estimate may differ from it by chance. Conversely, agreement does not establish that the external estimate is unbiased: offsetting sources of error can produce similar estimates.
Conditions for a Meaningful Comparison
Before calculating a discrepancy, specify the estimand: the outcome, time horizon, control treatment, and population to which the risk refers. For example, “response by week 16 among patients who met the trial’s eligibility criteria” is more precise than simply “response.” Endpoint ascertainment should be harmonized; a registry outcome recorded differently from a trial outcome may create a source difference unrelated to clinical risk.
Let \(S\) denote data source, with \(S=E\) for external and \(S=R\) for randomized control, and let \(X\) denote measured baseline covariates. A common approach models the binary outcome \(Y\) given source and covariates, then standardizes each source to a common covariate distribution. For a target population with covariate values \(X_1,\ldots,X_N\), the standardized risk for source \(s\) is the average of the model-predicted risks for those same individuals.
In Bayesian analysis, compute these standardized risks for each posterior draw. Subtract the randomized value from the external value within each draw to obtain a posterior distribution for \(\delta\). This preserves the uncertainty and any dependence induced by the joint outcome model. A credible interval for \(\delta\) describes posterior uncertainty about this model-based discrepancy, conditional on the data, model, and priors; it is not a frequentist confidence interval.
The interpretation requires adequate overlap in measured covariates, a sufficiently credible outcome model, and comparable outcome measurement. It also requires that the target population be defined consistently. If some trial patients have covariate profiles absent from the external data, model-based predictions there depend heavily on extrapolation. Covariate adjustment cannot correct unmeasured confounding or incompatible outcome ascertainment merely by including more variables.
Worked Benchmark With Covariate Standardization
Worked Example: Comparing Standardized Control Risks
Hypothetical data. A trial and an external cohort both evaluate response by week 16 under the same control regimen. The target is the trial population, which is divided into equal-sized lower-risk and higher-risk strata. A Bayesian model adjusts for additional prespecified baseline covariates within these strata. Under a uniform Beta(1,1) prior for each randomized-arm stratum risk, the randomized posterior means are \(25/52\) for lower-risk patients and \(17/52\) for higher-risk patients. The fitted external-control model gives posterior mean risks of 0.576923 and 0.431895, respectively, after standardizing to the corresponding trial strata. These are illustrative model outputs, not results from a real study.
The stratum weights are both 0.5, so the standardized randomized-control posterior mean is
The standardized external-control posterior mean is
Using the unrounded weighted values, the estimated discrepancy is
Thus the external model estimates a response risk about 10.06 percentage points higher than the randomized-control benchmark after standardization. This is a discrepancy to investigate, not evidence by itself that the external estimate is biased upward by exactly 10.06 percentage points. Posterior draws may also show substantial uncertainty around this difference.
Posterior Uncertainty and a Short Justification
A posterior mean discrepancy alone does not show whether the observed difference is precisely estimated. For a pre-specified tolerance \(c\), such as 0.05 on the risk-difference scale, useful summaries include a credible interval for \(\delta\) and \(P(|\delta|>c\mid\text{data})\). The latter is the posterior probability, under the specified model and data, that the discrepancy exceeds the chosen tolerance in magnitude. The tolerance should be clinically justified; it is not selected after seeing which value makes the comparison look favorable.
The calculation using posterior draws follows directly from the definition. For draw \(m\), calculate both standardized risks using the same target covariate values, then subtract. The empirical mean and quantiles of the resulting \(M\) values summarize the posterior distribution of the discrepancy.
For example, if 79 of 100 posterior draws have \(\delta^{(m)}>0.05\), then the estimated posterior probability that external risk exceeds randomized risk by more than five percentage points is 0.79. It is not the probability that a null hypothesis is true. If the two risks are estimated in separate models, draws can be paired independently only when that independence assumption is appropriate; a joint model should instead retain the posterior dependence between estimates.
Crude Versus Adjusted Comparisons
Worked Example: Why the Crude Gap Can Mislead
Hypothetical data. Suppose the trial population is 80% lower-risk and 20% higher-risk. Posterior mean response risks from the randomized arm are 0.50 and 0.30 in these strata; the external-control model gives corresponding risks of 0.60 and 0.40. The external cohort itself is 40% lower-risk and 60% higher-risk.
In the trial target population, the randomized risk is
Standardizing the external risks to that same population gives
The adjusted benchmark discrepancy is therefore \(0.56-0.46=0.10\), or 10 percentage points. In contrast, the external cohort’s crude risk is \(0.4(0.60)+0.6(0.40)=0.48\), and the trial’s crude risk is 0.46. The crude difference is only 2 percentage points because the external cohort contains more higher-risk patients. The crude comparison and the target-population comparison answer different questions; neither should be substituted for the other without stating its target.
Sensitivity Across Models and Data Sources
Benchmarking is most informative when the discrepancy is examined under defensible alternatives, rather than under a single preferred specification. For example, compare a minimally adjusted model with one including important prognostic covariates; compare alternative prespecified external databases or eligibility rules; and assess whether restricting to regions of covariate overlap changes the result. These are sensitivity analyses, not opportunities to choose whichever analysis gives the smallest discrepancy.
Worked Example: Interpreting Source and Adjustment Sensitivity
Hypothetical analysis. An analysis team reports posterior mean standardized response discrepancies relative to the same randomized benchmark. The following table gives posterior means and 95% credible intervals in percentage points.
| Analysis | Mean discrepancy | 95% credible interval |
|---|---|---|
| Primary source, prespecified adjustment | 10.1 | 1.2 to 19.0 |
| Primary source, reduced adjustment | 7.0 | -1.0 to 15.0 |
| Alternative eligible external source | 2.0 | -6.0 to 10.0 |
The primary adjusted result suggests a positive discrepancy, but its interval also includes small differences near zero. The reduced-adjustment result is less positive, while the alternative source is closer to the benchmark. This pattern indicates that the conclusion depends on adjustment and source choices. It does not establish which analysis is correct: differences might reflect population composition, residual confounding, endpoint recording, or random variation. The team should investigate those mechanisms, report the prespecified primary analysis, and show the alternatives transparently.
A narrow interval does not cure a poorly aligned data source, and a wide interval does not demonstrate compatibility. Precision and validity are separate concerns. In particular, if source-specific posterior uncertainty is understated by an overly restrictive model, the interval may look reassuring while the benchmarking exercise is not.
Alternatives, Diagnostics, and Limitations
Regression standardization is not the only way to align covariate distributions. Matching can restrict comparisons to similar patients but may discard data and change the target population. Weighting can target a chosen population but may become unstable when some patients receive extreme weights. A crude comparison is simple but does not adjust for measured case-mix differences. Whatever method is used, report the target population and inspect overlap and influential observations. The estimates should be compared on the same outcome and time horizon.
Warning signs include large changes in the discrepancy after modest covariate changes, external predictions far outside the observed data range, poor overlap, and notable differences across credible data sources. Check the outcome-model calibration and the outcome definitions before interpreting a discrepancy as a source effect. If a source contains few patients in a target-population stratum, its standardized estimate may be driven by extrapolation. Increasing model complexity does not automatically solve this problem.
The concurrent randomized arm is a useful reference, but it is not an infallible gold standard. Its estimate may be imprecise, and the trial’s selection criteria may limit how well it represents the intended target. Benchmarking also cannot diagnose every form of residual confounding: an unmeasured factor might affect both sources similarly, leaving the discrepancy small. Therefore, agreement is supportive evidence about compatibility under the analyzed conditions, not proof of exchangeability or absence of bias.
Common Pitfall
Correction: Closeness means the estimates are compatible to the observed precision under the chosen models, outcome definitions, and target population. It cannot rule out shared biases, unmeasured confounding, or coincidental agreement. Report the discrepancy and its uncertainty, and interpret it alongside diagnostics and sensitivity analyses.
Check Your Understanding
Answer each question, then compare with the solutions below.
- An external-control risk is standardized to a population with 70% lower-risk and 30% higher-risk patients. The stratum risks are 0.40 and 0.20. Calculate the standardized risk.
- A randomized benchmark risk is 0.35 and the standardized external risk is 0.42. Define the discrepancy and interpret its sign and size.
- Why can a crude external-control risk be a misleading benchmark when the source cohorts have different covariate distributions?
- A 95% credible interval for the discrepancy includes zero. Does this prove that the external and randomized risks are equal? Explain.
Solutions. (1) The standardized risk is \(0.7(0.40)+0.3(0.20)=0.28+0.06=0.34\), or 34%. (2) \(\delta=0.42-0.35=0.07\): the external estimate is 7 percentage points higher for the specified outcome and target population. (3) Different covariate mixtures can change crude risks even when stratum-specific risks are similar; standardization compares both sources using one common distribution. (4) No. The interval indicates uncertainty that includes zero under the model; it does not prove equality, and the interval’s width and clinically meaningful tolerances matter.