Common Errors in Nonconcurrent Control Analyses

Why Errors in Historical Comparisons Matter

Nonconcurrent controls can contribute information when a treatment arm enters a platform trial after the trial has begun. But patients treated in a later period were not randomized against patients enrolled earlier. Calendar time may bring changes in standard care, eligibility, outcome measurement, follow-up, or participating sites. An analysis can therefore produce a precise numerical contrast while leaving substantial uncertainty about what caused the difference.

The central discipline is to keep the design, analysis, and interpretation aligned. Define control status from trial participation and treatment availability at enrollment, examine whether the historical and treatment groups are comparable in relevant ways, and describe the resulting contrast as observational unless the design justifies a randomized interpretation.

Definition: A concurrent control is an eligible participant assigned to control during the period when the experimental arm is available for randomization in the trial. A nonconcurrent control is an eligible control participant enrolled in an earlier or otherwise nonoverlapping period, when that experimental arm was not available. Whether a participant is concurrent is determined by the enrollment or randomization period relative to arm availability—not by the date their outcome occurs or is measured.

Error 1: Defining Controls by Outcome Date

A common error is to call a control participant “concurrent” because their event, outcome assessment, or follow-up visit occurred during the same calendar dates as treatment-arm activity. This confuses the timing of an outcome with the timing of treatment assignment. A participant enrolled before an experimental arm opened did not become randomized against that arm merely because their later follow-up overlapped its recruitment period.

The distinction matters because an outcome date may itself depend on a participant’s health, follow-up schedule, or the event being studied. Selecting controls according to that date can create outcome-dependent inclusion or classification. Instead, establish the arm’s opening and closing dates and use each participant’s enrollment or randomization date, together with the trial’s eligibility and assignment records.

Worked Example: Outcome Dates Do Not Make Controls Concurrent

Suppose an experimental arm is available for randomization from May through August. Fifty control participants were randomized from January through April, before the arm opened. Ten of those controls have an event recorded from May through July. Fifty treatment participants were randomized from May through August, and 8 have an event recorded from August through October. An analyst labels the first ten controls “concurrent” because their events occurred while the experimental arm was recruiting, then reports the event-rate difference.

The apparent calculation is \(10/50-8/50=0.20-0.16=0.04\), or 4 percentage points more events among the selected controls. Reversing the subtraction to treatment minus control gives \(8/50-10/50=-0.04\), or 4 percentage points fewer events in the treatment group. Neither is a valid randomized contrast: all 50 controls were randomized before the arm opened. The fact that some outcomes occurred during recruitment does not change their assignment history.

The correct design classification is that the 50 earlier controls are nonconcurrent. The example does not provide a concurrent randomized control group for this arm. Any comparison using those controls must be presented and assessed as a historical observational comparison, with its assumptions and possible calendar-time biases made explicit.

Error 2: Ignoring Changes in Protocol or Standard Care

Historical comparisons are especially vulnerable when trial procedures or clinical practice change across periods. Relevant differences can include eligibility rules, dose or supportive-care policies, available standard treatments, sites participating, endpoint definitions, assessment schedules, and censoring or follow-up. A measured baseline covariate adjustment does not automatically repair a difference in how treatment or outcomes were defined.

Before pooling or comparing periods, create a period-by-period record of protocol versions and standard care. Ask whether the treatment strategies and endpoint have the same meaning across periods. If, for example, a later protocol uses a different outcome definition, a difference in observed event rates may reflect measurement rather than treatment. If standard care improves over time, historical controls may have a different prognosis even when recorded patient characteristics look similar.

Worked Example: A Change in Standard Care Alters the Contrast

Consider a hypothetical binary adverse outcome. In the treatment group, 20 of 160 participants experience the outcome. Among 120 earlier controls, 18 experience it. The treatment-minus-historical-control difference is

\frac{20}{160}-\frac{18}{120} =0.125-0.150 =-0.025,

so the observed proportions differ by -2.50 percentage points. The negative sign means the treatment group’s observed event proportion is lower by 2.50 percentage points; it does not establish that treatment caused the reduction.

Now suppose standard care changed after those 120 controls were enrolled. In a later period, 11 of 100 control participants experienced the same defined outcome. The later control proportion is \(11/100=0.110\), or 11.0%. Comparing treatment with this later control group gives \(0.125-0.110=0.015\), or 1.50 percentage points more events in the treatment group. If the earlier and later controls are pooled, the historical-control proportion is \((18+11)/(120+100)=29/220=0.1318\); the treatment-minus-pooled difference is \(0.125-0.1318=-0.0068\), or about -0.682 percentage points.

These three contrasts differ because the control periods differ; the arithmetic alone cannot determine which comparison is causally relevant. The analyst must investigate the care change and explain why any chosen historical period is appropriate. Pooling periods without justification can conceal clinically important drift.

Error 3: Extrapolating Beyond Covariate Overlap

Adjustment can only compare treatment and control participants in ways supported by the observed data. For a covariate pattern represented among treated participants but absent among controls—or vice versa—the adjusted comparison relies on a model’s extrapolation rather than an observed within-pattern contrast. This is a positivity, or overlap, problem. A sophisticated regression model cannot manufacture information about the missing comparison group.

For an adjusted causal contrast to have the usual interpretation, relevant covariates must be measured adequately; treatment and control outcomes must be comparable conditional on those covariates; and each covariate pattern in the target population must have a positive chance of belonging to either group. In nonconcurrent comparisons, calendar period can be particularly problematic: if treatment is only given in the later period and controls only in the earlier period, there may be no within-period treatment-control comparison. Adjustment cannot separate treatment from period without additional design information or assumptions.

Worked Example: A Missing Severity Group Prevents Direct Comparison

Suppose a historical control group contains 80 participants, all with mild disease, while a later treatment group contains 80 participants, all with severe disease. The observed event proportions are 8/80=10% in controls and 16/80=20% in the treatment group. The crude treatment-minus-control difference is \(0.20-0.10=0.10\), or 10 percentage points.

But there are no severe historical controls and no mild treatment participants. Thus, for the severe stratum there is no observed control outcome, and for the mild stratum there is no observed treatment outcome. A model that reports an adjusted comparison for a population containing both severity groups must extrapolate at least one of these missing outcomes. The 10-point difference cannot distinguish a treatment effect from the difference in disease severity. Without added design or credible external information, these data do not support a direct severity-adjusted treatment contrast.

Error 4: Calling an Adjusted Historical Contrast Randomized

Regression, matching, weighting, or standardization may adjust for measured differences. They do not retroactively randomize historical participants. Random assignment balances prognostic factors in expectation within the randomized comparison; adjustment in an observational comparison instead depends on measured covariates, model choices, adequate overlap, and the absence of important unmeasured confounding. Calendar-time differences can remain even after a fitted model includes some period indicators.

The standardization identity makes the assumptions visible. Let \(A=1\) denote treatment and \(A=0\) control, \(Y^a\) the outcome that would occur under assignment \(a\), and \(X\) measured covariates. For discrete \(X\), a standardized risk difference can represent an average causal contrast if consistency holds, there is conditional exchangeability \((Y^1,Y^0)\) independent of \(A\) given \(X\), and positivity holds for every covariate pattern in the target population:

E(Y^1-Y^0) = \sum_x \left\{ E(Y\mid A=1,X=x)-E(Y\mid A=0,X=x) \right\}P(X=x).

The formula uses observed conditional means to stand in for the unobserved potential-outcome means only under those assumptions. In a nonconcurrent comparison, conditional exchangeability is not guaranteed by randomization between historical controls and the later treatment group. Unmeasured changes over time can violate it, and lack of overlap can make the required conditional means unavailable.

Worked Example: Standardization Is Not Randomization

Suppose a hypothetical analysis has two severity strata, with observations in both groups. Among 60 treated participants with mild disease, 6 have the event; among 40 treated participants with severe disease, 12 have it. Among 80 historical controls with mild disease, 8 have the event; among 80 historical controls with severe disease, 20 have it. Standardize both groups to a target population with 50% mild and 50% severe disease.

The treated risks are \(6/60=0.10\) for mild disease and \(12/40=0.30\) for severe disease. The standardized treated risk is \(0.50(0.10)+0.50(0.30)=0.20\), or 20.0%.

The historical-control risks are \(8/80=0.10\) for mild disease and \(20/80=0.25\) for severe disease. The standardized historical-control risk is \(0.50(0.10)+0.50(0.25)=0.175\), or 17.5%. The standardized treatment-minus-control contrast is \(0.20-0.175=0.025\), or 2.50 percentage points more events in the treated group.

This calculation adjusts the severity distribution, but it does not establish that treatment caused the difference. For a causal interpretation, severity must be sufficient to control confounding, the outcome and treatment versions must be comparable across periods, and other assumptions in the standardization identity must hold. An adjusted result should therefore be described as an adjusted comparison with historical controls unless its causal assumptions are justified.

Diagnostics, Alternatives, and Limits

A defensible analysis makes its vulnerabilities visible rather than relying on a single adjusted estimate. Useful checks include:

Tabulate participant enrollment dates against the experimental arm’s opening and closing dates; verify that concurrent status is not based on outcome dates.
Compare protocol versions, eligibility, endpoint definitions, follow-up windows, standard care, sites, and regions across periods.
Inspect covariate distributions and the number of participants from each group across clinically important covariate strata. Sparse or empty cells flag extrapolation.
Check whether conclusions change when plausible historical periods or model specifications are used. Treat such sensitivity analyses as assessments of robustness, not proof that unmeasured confounding is absent.
Report the estimate, its uncertainty, the target population, the control period used, and the assumptions needed for a causal interpretation.

The main alternative, when available, is a concurrent randomized control comparison for the arm. It directly preserves the randomized contrast for participants assigned while both groups are available, though it may be less precise than an analysis augmented with historical data. Historical controls may add precision under suitable assumptions, but the added sample size does not remove bias. Restriction to a population with adequate overlap, a carefully justified adjustment, or separate presentation of concurrent and nonconcurrent estimates may be more credible than a single pooled comparison. Each choice changes the target population or relies on assumptions that should be stated.

Important limitations remain even after careful analysis. Some temporal differences are not recorded; measured variables may not capture changes in clinical decision-making; and a large sample can make a biased contrast look highly precise. Confidence intervals quantify sampling uncertainty under the analysis model, not uncertainty from every source of bias. In particular, a narrow interval around a historical comparison is not evidence that the treatment and control groups were exchangeable.

Common Mistake and Correction

Common mistake: “We adjusted for age and severity, so the historical-control comparison is randomized and any remaining difference is the treatment effect.”

Correction: Adjustment controls only the measured differences represented adequately in the model. It does not create random assignment, resolve protocol or standard-care changes, or supply comparisons where covariate overlap is absent. Describe the estimate as an adjusted observational contrast and state the assumptions required for a causal interpretation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

An earlier control participant is followed up during the months when a new arm is recruiting. Is that control concurrent? Explain.
A treatment group has 24 events among 200 participants and a historical control group has 30 among 200. Calculate the treatment-minus-control difference in event proportions and state what it does—and does not—show.
Why can a covariate-adjusted model fail when all severe participants received treatment and all mild participants were controls?
Name two kinds of changes across protocol periods that can undermine a historical comparison even if the baseline covariates are balanced.
What additional assumptions are needed before an adjusted historical contrast can be interpreted causally?

Solutions. (1) No. Concurrent status follows enrollment or randomization relative to the arm’s availability; later follow-up does not change the earlier assignment period. (2) The treatment proportion is \(24/200=0.12\); the control proportion is \(30/200=0.15\). Thus the difference is \(0.12-0.15=-0.03\), or 3 percentage points fewer observed events in the treatment group. It is a sample contrast, not by itself a causal treatment effect. (3) There is no treatment-control overlap within severity groups, so the model must extrapolate missing outcomes; treatment and severity cannot be separated from these data alone. (4) Examples include changing standard care, eligibility criteria, endpoint definitions, follow-up schedules, or participating sites. (5) At minimum, the comparison requires consistency, conditional exchangeability given measured covariates, positivity in the target population, and sufficiently accurate measurement and modeling of the variables used to justify those assumptions.