Why Reporting Must Make the Design Visible
An external-control estimate is interpretable only when readers can see who the estimate concerns, where the comparison data came from, how the cohorts were made comparable, and which assumptions remain unverified. A posterior distribution can describe uncertainty under a statistical model; it cannot by itself establish that trial participants and external controls were exchangeable, or remove bias caused by differences in care, eligibility, or outcome measurement.
This tutorial focuses on reporting those elements alongside the treatment estimate. The goal is not to replace the detailed analysis in the earlier material on transport weights, endpoint harmonization, and posterior predictive checks. It is to make the evidence auditable: a reader should be able to follow the path from the target question to the reported estimate and understand what would change its interpretation.
What a Complete Report Identifies
Begin with the clinical question, not the data source. State the target population using operational eligibility criteria, the treatment strategy, the external-control strategy, the endpoint, and its time horizon. For example, “the 12-month risk difference in all-cause mortality for eligible adults receiving the investigational treatment versus comparable adults receiving usual care” is more informative than “the treatment effect.” State the direction of subtraction and units: a risk difference of \(0.03\) is a 3-percentage-point higher risk in the treatment group if the contrast is treatment minus control.
Then identify how the source data were generated and selected. Report the registry or database, calendar period, sites or regions, care setting, eligibility criteria, index date, follow-up rules, and endpoint ascertainment. Give the number of records screened and the count excluded at each material step, with reasons. Clarify whether exclusions were determined before examining outcomes. Describe missingness in important baseline covariates and outcomes, and explain how it was handled.
The cohort description should make differences in data collection visible. Trial and external records may use different visit schedules, coding systems, laboratory methods, or definitions of treatment initiation. State how variables and endpoints were harmonized, and which discrepancies could not be resolved. Where relevant, explain whether external patients were eligible for the trial at the same index time and whether their follow-up began under a comparable rule.
For adjustment, report the covariates included, why they were selected, how they were measured, and the method used to account for them. State whether the reported contrast is standardized to the target population or refers to another population. Explain how missing covariates and extreme or unstable weights were handled. A list of covariates is not a substitute for a rationale: clinical prognostic factors, treatment-selection factors, and variables affected by treatment have different roles.
Worked Example: Describing Cohort Construction
Suppose a hypothetical trial enrolls adults with a specified rare disease and compares an investigational therapy with an external registry cohort. Investigators identify 340 potentially eligible registry records. Using prespecified, sequential criteria, they exclude 30 records that fail the disease-stage criteria, 22 without the required baseline covariates, 18 without adequate 12-month outcome ascertainment, and 10 duplicate records. The analysis cohort therefore contains \(340-30-22-18-10=260\) external controls.
A clear report would state the target population as the trial-eligible adults meeting the specified disease-stage criteria at the trial’s index date, rather than simply “registry patients.” It would report the four exclusion counts and their reasons, the 260 retained controls, the registry calendar period, and how baseline and outcome definitions were aligned with the trial. It would also identify whether the 22 records missing baseline covariates were excluded, imputed, or otherwise handled; in this example they were excluded, so the report should discuss the possibility that the retained cohort differs from those records.
These counts describe cohort construction, not representativeness. The registry may still differ from the target population on measured or unmeasured factors. The report should therefore present the adjustment and diagnostic evidence separately rather than treating the final sample size as evidence of comparability.
Reporting the Estimate and Its Uncertainty
For a Bayesian analysis, report the treatment contrast on a clinically meaningful scale, its posterior summary, and a credible interval with its level. Identify the likelihood, prior distributions relevant to the estimate, adjustment model, and computational method sufficiently for another analyst to reproduce the analysis. Report posterior probabilities only for explicit events and thresholds, such as the probability that the risk difference is below zero or below a prespecified clinically important margin. A credible interval is a summary of the posterior distribution conditional on the model and its assumptions; it is not a measure of uncertainty about whether those assumptions are true.
Also distinguish the modeled uncertainty from design limitations. A narrow posterior interval does not account automatically for unmeasured confounding, selection into the registry, outcome misclassification, or lack of overlap. State whether posterior summaries include uncertainty in estimated adjustment or standardization quantities. If a calculation combines independent trial and external estimates, justify the independence assumption; if they share estimated parameters or data, account for the induced covariance rather than simply adding variances.
Worked Example: Reporting a Standardized Risk Difference
Consider a hypothetical analysis with two prespecified strata, weighted \(0.60\) and \(0.40\) to represent the target population. The external-control posterior mean risks are \(11/102\) and \(41/102\) in the two strata. The trial-treatment posterior mean risk is \(0.2600\). Suppose the fitted model gives trial stratum risk variances \(0.003915\) and \(0.005761\), and the posterior variance of the standardized external-control risk is \(0.000709697\). These variance values are stipulated model outputs for this example, not quantities inferred from the two displayed external means alone.
The standardized external-control risk is
Define the risk difference as trial treatment minus external control. Its posterior mean is therefore
Thus the posterior mean risk is about 3.451 percentage points higher in the trial-treatment group. The sign alone does not establish a causal increase; interpretation depends on the comparability assumptions and design.
Assuming the two trial stratum-specific risks are posterior-independent, the trial standardized variance, using the stated stratum variances and weights, is
If the trial and external posterior estimates are independent conditional on the model, the variance of their difference is the sum of their variances:
The corresponding standard deviation is \(\sqrt{0.003040857}\approx0.05514\). Under a normal posterior approximation, a 95% credible interval is
In context, this approximate interval ranges from a 7.36-percentage-point lower risk to a 14.26-percentage-point higher risk in the trial-treatment group. It is wide and includes zero. A report should label it as a Bayesian credible interval under the stated model and note the approximation; it should not present it as resolving the external-control design assumptions.
Diagnostics and Sensitivity Analyses
Diagnostics should answer whether important parts of the analysis are behaving as intended. Report covariate balance before and after adjustment, overlap in the covariate distributions, and any extreme weights or influential observations. Describe checks of endpoint definitions, follow-up completeness, and model fit. Posterior predictive checks can identify patterns the fitted model fails to reproduce, but passing a check does not demonstrate absence of unmeasured confounding or selection bias.
Sensitivity analyses should vary plausible analytic choices that could materially affect the estimate. Examples include alternative eligibility or index-date rules, reasonable endpoint definitions, different adjustment specifications, approaches to missing data, and restrictions to regions of adequate covariate overlap. If unmeasured-confounding analyses are included, state the assumed strength and direction of the unmeasured factor and how it changes the estimate. Present the results side by side, explain which choices were prespecified, and avoid selecting one result solely because it is more favorable.
Worked Example: Summarizing Sensitivity Analyses
Suppose a hypothetical primary analysis estimates a 12-month risk difference of \(0.035\), treatment minus external control, with a 95% posterior credible interval of \((-0.074, 0.143)\). Analysts also conduct three prespecified sensitivity analyses. The values below are hypothetical posterior summaries from those alternative specifications.
| Analysis | Risk difference | 95% credible interval | Change from primary analysis |
|---|---|---|---|
| Primary adjustment | 0.035 | (−0.074, 0.143) | — |
| Restrict to covariate overlap | 0.018 | (−0.091, 0.127) | Estimate shifts 1.7 percentage points lower |
| Alternative outcome-missingness assumption | 0.061 | (−0.051, 0.173) | Estimate shifts 2.6 percentage points higher |
| Alternative adjustment specification | 0.047 | (−0.066, 0.158) | Estimate shifts 1.2 percentage points higher |
A suitable interpretation is that the posterior mean risk difference remains positive across these specifications, while all four intervals include zero and allow for effects in either direction. The mean changes from 0.018 to 0.061 across the sensitivity analyses, so the point estimate is not invariant to modeling choices. These results do not prove that the analyses cover every source of bias; they show how selected, stated assumptions affect the reported comparison. The target population should also be revisited for the overlap-restricted analysis, because restricting the data may change the population to which that estimate applies.
Assumptions, Alternatives, and Limitations
A causal interpretation requires assumptions beyond correct arithmetic and adequate model fit. In particular, after adjustment for measured covariates, the treatment groups must be exchangeable with respect to potential outcomes; relevant covariates must be measured adequately; comparable patients must have a positive chance of appearing in each group; and treatment and outcome definitions must be sufficiently consistent across sources. There must also be no relevant interference between individuals. These conditions are substantive and cannot be established solely by balance summaries or posterior diagnostics.
A concurrent randomized control arm is generally a stronger basis for a causal comparison because randomization supports exchangeability in expectation. An external control may be useful when a concurrent arm is infeasible or when additional context is valuable, but it relies on stronger comparability assumptions. When a concurrent randomized arm is available, an external cohort can provide supporting information, but the report should make clear which contrast is randomized and which depends on external data. A single-arm estimate without an external comparator answers a different question and should not be presented as though it were a controlled treatment contrast.
Limitations should be specific: for example, unmeasured disease severity, different referral patterns, calendar-time changes in usual care, incomplete outcome capture, or limited overlap. State their possible direction when it can be reasoned about, but do not assert a direction without justification. Keep the target population and interpretation aligned with what the data can support. In particular, transport or weighting may improve alignment on measured characteristics but does not correct discrepancies in unmeasured factors or endpoint ascertainment.
A Practical Reporting Sequence
State the target population, treatment strategies, endpoint, time horizon, contrast direction, and estimand.
Name the external source, calendar period, eligibility criteria, index date, follow-up, exclusions, and endpoint harmonization.
Describe adjustment, priors and model components relevant to the estimate, missing-data handling, and computation.
Report the estimate in context, its credible interval, and any posterior probability with the event and threshold stated.
Summarize diagnostics, sensitivity analyses, alternative approaches, and limitations that affect interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions below.
- Why is “registry patients” insufficient as a description of the target population?
- A report gives a posterior risk difference of \(-0.04\), treatment minus external control. Interpret its direction and identify one reason this need not establish a causal benefit.
- What does a 95% posterior credible interval communicate, and what important uncertainty may it fail to include?
- Why can restricting to adequate covariate overlap change the meaning of the estimand?
- Name two diagnostics and one sensitivity analysis that would help readers assess an external-control comparison.
Solutions. (1) It does not specify eligibility, disease stage, index date, or the population the result is intended to represent. (2) The posterior mean risk is 4 percentage points lower in the treatment group; unmeasured confounding or noncomparable outcome ascertainment could still explain some or all of the difference. (3) Conditional on the stated model and data, the interval contains 95% of the posterior probability for the parameter; it may not include uncertainty from unmeasured confounding or misspecified data-generating assumptions. (4) The restricted patients may represent a narrower population than the original target, so the estimate may no longer apply to all eligible patients. (5) Examples include post-adjustment covariate balance, overlap, extreme-weight checks, and posterior predictive checks; a sensitivity analysis could vary the missing-outcome assumption, index-date rule, or adjustment specification.