Why Acceptability Is a Design Argument
A treatment group in a clinical trial may be compared with patients who were not randomized alongside it. That comparison can be useful, especially when randomization is infeasible or a concurrent control raises ethical or practical concerns. But it is not equivalent to a randomized comparison: differences between the groups may reflect who entered each data source, when they were treated, or how their outcomes were measured, rather than an effect of treatment.
The regulatory argument for an external control therefore starts with its intended role. It should identify the question the comparison is meant to answer, why an external control is appropriate for that question, and what assumptions would be needed to interpret the results. A persuasive argument does not claim that a statistical adjustment makes the groups interchangeable. It explains why the comparison is credible in context, what uncertainties remain, and how those uncertainties limit the conclusion.
An external control can play different roles: it may be the principal comparator in a single-arm trial, provide contextual evidence alongside other evidence, or contribute information to a Bayesian analysis. These roles should not be blurred. In particular, using external observations as comparator data is not the same as merely specifying a prior distribution. The planned analysis must make clear which patients’ outcomes are being compared and what each source contributes to the regulatory question.
State the Intended Comparison Before Assessing It
A useful design argument describes a hypothetical target comparison: which patients would be eligible, what treatment strategies would be compared, when follow-up would begin, what outcome would be measured, and over what period. This makes it possible to judge whether the trial participants and external controls could stand in for the groups in that comparison. Without a clearly stated target, the phrase “similar patients” is too vague to evaluate.
The case for an external control is usually stronger when randomization is not feasible or acceptable, the disease is rare or severe, the treatment effect is expected to be substantial, and the natural history or usual-care outcomes are sufficiently well characterized. These considerations do not by themselves establish comparability. A serious unmet need can explain why an external control is considered; it cannot show that the external patients provide an unbiased estimate of what would have happened to the trial participants without treatment.
The argument should also explain why the chosen external source is appropriate for its role without treating source selection as a box-checking exercise. At this stage, the key issue is whether the proposed comparison is coherent: the comparator should represent a clinically meaningful alternative for the trial’s target population, and the design should make clear how each group enters follow-up and how outcomes will be interpreted. Detailed evaluation of the data source itself is a separate task.
Worked Example: Defining the Role of an External Control
Suppose a hypothetical single-arm trial evaluates a new therapy for a rapidly progressive, rare disease. Investigators propose comparing one-year survival in trial participants with survival in patients treated with usual care in a national registry. The registry is not randomized, and the treatment trial begins several years after most registry patients were treated.
Question and role. The comparison is intended to inform whether one-year survival under the new therapy is better than survival under usual care for patients eligible for the trial. The registry is a proposed external control, not a randomized control and not simply a prior distribution.
Why consider this design? The rarity and severity of the disease may make a randomized control difficult to implement. These facts support considering an external control, but they do not establish that registry outcomes represent the trial participants’ counterfactual outcomes under usual care.
Key comparability concern. The time gap could matter if supportive care, diagnostic practice, or eligibility criteria changed. The argument should explain whether the registry patients and trial participants can be aligned on disease stage, treatment history, and the start of outcome follow-up. If those conditions are not credible, the registry may provide context but not a defensible primary comparator.
Conclusion. The proposed role is clear, but acceptability remains conditional on a plausible comparison and candid treatment of calendar-time differences. Rarity alone is not enough to justify interpreting the observed survival difference as a treatment effect.
Comparability and the Assumptions Behind Interpretation
Comparability is broader than matching a few baseline summaries. The trial and external groups should be considered with respect to the factors that can affect both treatment assignment or data-source membership and the outcome. Examples include disease severity, prior treatment, comorbidity, age, access to care, and calendar time. Outcome definitions, assessment schedules, follow-up start, and censoring also matter: groups cannot be meaningfully compared if “one-year survival” is constructed differently across sources.
A causal interpretation requires assumptions that are stronger than observing similar measured characteristics. Let \(A=1\) indicate trial treatment and \(A=0\) indicate usual care, \(Y\) be a binary outcome by a prespecified time, and \(X\) be measured baseline characteristics. Let \(S\) identify the source, such as the trial or external data. Consistency requires that the observed outcome under the treatment actually received corresponds to the relevant potential outcome. Conditional exchangeability requires that, within levels of \(X\), treatment and source membership do not leave important differences in potential outcomes. Positivity requires that patients with relevant \(X\) values in the target trial population have comparable external controls. Transportability requires that, given \(X\), the external controls’ outcomes under usual care represent the outcomes that the target trial population would have had under usual care.
The standardization formula illustrates the logic. For discrete baseline characteristics \(X\), the external data estimate the comparator outcome within each covariate group; those risks are then weighted to the covariate distribution of the target trial population. The formula follows from the law of total probability. Under conditional exchangeability and transportability, the external conditional risk equals the target population’s counterfactual comparator risk within each \(X\) group. Averaging those conditional risks over the target population gives its overall comparator risk:
This calculation does not remove the assumptions; it makes them visible. If an important prognostic factor is unmeasured, if some trial participants have no comparable external controls, or if outcomes are defined differently, the standardized number may still be biased. A regulatory argument should identify the variables and design features that make the assumptions plausible and acknowledge those that cannot be established from the data.
Worked Example: Describing Baseline Imbalance
In a hypothetical comparison, trial participants have mean age 54 years with standard deviation 12 years; external controls have mean age 60 years with standard deviation 10 years. A descriptive standardized mean difference (SMD) is the difference in means divided by the square root of the average of the two variances:
The negative sign indicates that the trial group is younger on average; the absolute difference is about 0.543 pooled standard deviations. This is a description of an observed imbalance, not a test of whether the groups are exchangeable. There is no universal SMD cutoff that establishes acceptability. The clinical importance of age, its relationship to the outcome, and whether the analysis can support the needed comparison all require consideration.
Suppose a binary severity indicator is present in 40% of trial participants and 25% of external controls. Its descriptive SMD is:
This also signals a difference in measured baseline composition. Neither SMD proves that adjustment will succeed: unmeasured severity differences, lack of overlap, or differences in treatment era could remain.
Why Crude Comparisons Can Mislead
The following example shows how the external group’s composition can change the apparent comparison. For simplicity, assume that the outcome is assessed at the same time from a common follow-up origin, and that the two groups have the same definitions and ascertainment. Even with those favorable conditions, the groups differ in disease severity.
Worked Example: A Different Patient Mix Changes the Estimate
In a hypothetical trial, 80 treated participants include 40 with severe disease and 40 with less severe disease. There are 20 outcomes among the severe participants and 28 among the less severe participants, for 48 outcomes overall. The external group has 80 usual-care patients: 60 with severe disease, of whom 16 have the outcome, and 20 with less severe disease, of whom 18 have the outcome.
The treated group’s crude outcome risk is \(48/80=0.600\), or 60.0%. The external group’s crude risk is \((16+18)/80=0.425\), or 42.5%. The crude risk difference, treated minus external, is:
That is a crude difference of 17.5 percentage points. It does not account for the different proportions with severe disease. Standardizing the external risk to the trial’s 50% severe and 50% less-severe composition gives:
The trial risk remains 60.0% in this simple stratification. The standardized difference is therefore \(0.600-0.5833=0.0167\), or about 1.67 percentage points. The crude and standardized differences differ substantially because the external group contains more severe patients, while the observed risks also vary across severity strata.
This calculation illustrates the role of measured composition; it does not establish a treatment effect. A causal reading would additionally require adequate adjustment for all relevant prognostic differences, adequate overlap, and credible outcome and follow-up alignment. The example’s small stratified data also show why a numerical adjustment should be interpreted with attention to uncertainty and the assumptions it depends on.
Diagnostics, Alternatives, and Limits
Before presenting an external comparison as decision-relevant, examine whether the argument is failing in ways that cannot be fixed by a model. Warning signs include important baseline differences with no plausible adjustment strategy; trial participants whose characteristics are absent from the external data; calendar-time changes in care; different eligibility or outcome definitions; different follow-up origins; and outcome ascertainment that depends on the data source. A large sample does not repair these design problems. It can make a biased estimate more precise without making it more credible.
The principal alternative is a concurrent randomized control. Randomization supports baseline comparability in expectation and usually provides a clearer basis for causal interpretation, though it may be infeasible, unethical, or poorly matched to an urgent clinical context. An external control may be more appropriate in some settings, but depends on stronger assumptions about comparability and transportability. A single-arm study without an external comparator may describe outcomes under treatment but does not, by itself, estimate what would have happened under usual care. Historical benchmarks can provide context, but comparisons against them require similar caution and should not be presented as equivalent to a randomized contrast.
An external control can also remain informative when causal claims are limited. It may help characterize disease course, place observed outcomes in context, or support an evidence package alongside other evidence. The strength of the conclusion should match the strength of the design. Avoid presenting a favorable observed contrast as definitive proof of treatment benefit when source differences could plausibly explain it.
Name the target population, treatment and comparator, outcome, and follow-up period.
Describe the design context without treating practical or ethical constraints as evidence of comparability.
Address baseline factors, overlap, treatment context, outcome definitions, and follow-up alignment.
Identify residual uncertainty and distinguish causal evidence from contextual or supportive evidence.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What makes a control group “external,” and how does it differ from a concurrent randomized control?
- A trial group and an external group have identical mean ages. Does this establish that an adjusted external comparison is unbiased? Why or why not?
- In the stratified example, what was the crude risk difference, and what was the standardized risk difference?
- Name two design features, other than baseline covariates, that can undermine comparability.
- Why can a large external dataset still give weak causal evidence?
Solutions. (1) An external control consists of patients outside the trial’s randomized treatment comparison whose outcomes serve as a comparator. A concurrent randomized control is assigned within the trial under its randomization scheme. (2) No. Equal mean age does not establish balance in other prognostic factors, conditional exchangeability, adequate overlap, or transportability. (3) The crude difference was \(0.600-0.425=0.175\), or 17.5 percentage points; standardizing the external risk to the trial’s severity mix gave \(0.600-0.5833=0.0167\), or about 1.67 percentage points. (4) Examples include different calendar periods, outcome definitions, follow-up origins, and ascertainment procedures. (5) A large sample can reduce sampling variability but cannot, by itself, correct confounding, source differences, or a mismatch between the external group and the target population.