Site and Regional Variation in Nonconcurrent Controls

Why Geographic Composition Matters

A historical control group can differ from a current treatment group for two separate reasons: patients may have been enrolled at different calendar times, and they may have come from different sites or regions. These mechanisms are not interchangeable. Calendar time can capture changes in care or outcome measurement, while site and region can reflect differences in patient mix, referral patterns, clinical practice, or baseline risk. A pooled comparison can combine all of these differences with the treatment contrast.

The key question in this tutorial is narrower than a complete analysis of temporal bias: does the comparison have adequate geographic and site-level comparability? Site adjustment may reduce bias from differing site composition, but it cannot manufacture information for sites or regions represented in only one group. Nor does adjustment for site, by itself, remove calendar-time differences between historical controls and current patients.

Definition: Let \(S\) denote a patient's site, \(R\) the patient's region, \(P\) the calendar period, \(T\) the treatment indicator, and \(Y\) the outcome. A site-comparable contrast compares treatment and control outcomes after placing their site distributions on a common target distribution. Geographic overlap means that the sites or regions relevant to that target are represented sufficiently in both groups to support the comparison.

The outcome \(Y\) may be continuous, binary, or time-to-event; the examples below use binary outcomes so that differences can be read in percentage points. The same composition logic applies to other outcome scales, although the appropriate model and effect measure depend on the outcome.

Decomposing a Crude Contrast

Suppose there are \(K\) sites. Let \(\mu_{ts}\) be the mean outcome among group \(t\) at site \(s\), where \(t=1\) denotes treatment and \(t=0\) denotes control. Let \(w_{ts}\) be the proportion of group \(t\) enrolled at site \(s\). The crude group mean is the site-specific mean averaged using that group's own site proportions:

\bar{\mu}_t=\sum_{s=1}^{K}w_{ts}\mu_{ts}, \qquad \sum_{s=1}^{K}w_{ts}=1.

If the site proportions differ, \(\bar{\mu}_1-\bar{\mu}_0\) is not generally an average of within-site contrasts using a common set of weights. One useful decomposition, taking the control site distribution as the reference, is:

\bar{\mu}_1-\bar{\mu}_0 = \sum_{s=1}^{K}w_{0s}(\mu_{1s}-\mu_{0s}) + \sum_{s=1}^{K}(w_{1s}-w_{0s})\mu_{1s}.

The first term is a weighted average of within-site differences, using the control distribution. The second is a composition term: it reflects the difference in site weights applied to the treatment-group site means. This decomposition is descriptive. For a causal interpretation, the within-site contrasts must also be protected from temporal and other confounding.

Worked Example: A Crude Difference Driven by Site Mix

Consider a hypothetical binary outcome. Historical controls include 90 patients from Site A, of whom 9 have the outcome, and 10 from Site B, of whom 3 have it. Current treated patients include 50 from Site A, of whom 6 have the outcome, and 150 from Site B, of whom 48 have it. Compare the crude contrast with an equally weighted site-standardized contrast.

At Site A, the control risk is \(9/90=0.10\), and the treatment risk is \(6/50=0.12\). The within-site risk difference is \(0.12-0.10=0.02\), or 2 percentage points. At Site B, the risks are \(3/10=0.30\) and \(48/150=0.32\), so the risk difference is again \(0.02\).

The pooled control risk is \((9+3)/(90+10)=12/100=0.12\). The pooled treatment risk is \((6+48)/(50+150)=54/200=0.27\). Thus the crude difference is \(0.27-0.12=0.15\), or 15 percentage points.

For a target population with equal weight at each site, standardize each group to weights \(0.5\) and \(0.5\). The standardized control risk is \(0.5(0.10)+0.5(0.30)=0.20\). The standardized treatment risk is \(0.5(0.12)+0.5(0.32)=0.22\). The standardized difference is \(0.22-0.20=0.02\), or 2 percentage points.

The large crude contrast arises because 90% of controls came from the lower-risk Site A, whereas 75% of treated patients came from the higher-risk Site B. Standardization removes this particular site-composition difference. It does not establish that the remaining 2-point difference is causal: the groups are from different periods, so temporal differences still need consideration.

Standardization and the Conditions It Needs

A clear way to report a site-adjusted comparison is to specify the target site distribution. Let \(q_s\) be the target proportion at site \(s\), with \(q_s\geq 0\) and \(\sum_s q_s=1\). The standardized mean for group \(t\) is:

\mu_t^{(q)}=\sum_{s=1}^{K}q_s\mu_{ts}, \qquad \Delta^{(q)}=\mu_1^{(q)}-\mu_0^{(q)} =\sum_{s=1}^{K}q_s(\mu_{1s}-\mu_{0s}).

The target weights might represent the current treatment population, a planned recruitment population, or an explicitly chosen equal-site target. These choices answer different questions. The weights should be named and justified rather than selected after seeing which produces the most favorable result.

For a causal interpretation of \(\Delta^{(q)}\), a potential-outcomes formulation makes the assumptions explicit. Let \(Y(1)\) and \(Y(0)\) denote the outcomes that a patient would have under treatment and control, respectively. Within the target sites, the observed site-specific contrast must identify the corresponding potential-outcome contrast. This requires, at minimum, consistent outcome and treatment definitions; no relevant unmeasured confounding after conditioning on the variables used in the analysis; and adequate representation of both treatment conditions in the target strata. With nonconcurrent controls, the no-unmeasured-confounding condition must include calendar-time factors that affect outcome and treatment availability.

In practice, treatment availability may be determined by period: historical patients are controls and current patients receive treatment. If the analysis conditions on exact calendar period and no period contains both treatment groups, there is no within-period treatment comparison. A model that includes unrestricted period effects and a treatment indicator cannot separate them in that setting. A treatment estimate can then depend on modeling or extrapolation assumptions about time trends, not merely on adding period and site terms to a regression.

Key condition: Site overlap is necessary for direct site-standardized comparisons at the target sites, but it is not sufficient for causal identification. The historical and current groups must also be comparable with respect to important time-varying factors, or the analysis must make and defend assumptions about their effects.

Worked Example: Standardizing to a Stated Target

Suppose a hypothetical analysis has three sites. Historical control and current treatment outcome means are as follows: Site A, 10% and 14%; Site B, 20% and 22%; Site C, 30% and 30%. The target population is specified in advance to have weights 0.50, 0.30, and 0.20 at A, B, and C, respectively. Calculate the standardized contrast.

The standardized control mean is \(0.50(10\%)+0.30(20\%)+0.20(30\%)=5\%+6\%+6\%=17\%\). The standardized treatment mean is \(0.50(14\%)+0.30(22\%)+0.20(30\%)=7\%+6.6\%+6\%=19.6\%\).

The standardized risk difference is \(19.6\%-17.0\%=2.6\) percentage points. Equivalently, the site-specific differences are 4, 2, and 0 percentage points, and their target-weighted average is \(0.50(4)+0.30(2)+0.20(0)=2.6\) percentage points.

This calculation is supported descriptively because both groups have data at each target site. A causal interpretation still requires the historical-versus-current comparison to be credible within sites, including adequate handling of calendar-time differences. The standardized result applies to the stated target weights, not automatically to every possible population.

Separating Site Effects from Calendar-Time Effects

A useful organizing model for a continuous outcome is:

Y_i=\beta_0+\tau T_i+\gamma_{P_i}+\delta_{S_i}+\boldsymbol{\eta}^{\mathsf T}\mathbf{X}_i+\varepsilon_i,

where \(\gamma_{P_i}\) represents period differences, \(\delta_{S_i}\) represents site differences, \(\mathbf{X}_i\) contains measured patient characteristics, and \(\varepsilon_i\) is the residual. This is a model specification, not a guarantee of unbiasedness. It assumes that the chosen functional form adequately describes relevant differences; the displayed additive form also assumes no treatment-by-site or treatment-by-period interaction unless such terms are added.

Site and period can be difficult to disentangle. A site may enter the study only in later periods, or an entire region may contribute only historical controls. If site and period are strongly associated, estimates of their separate effects can be imprecise. If treatment status is determined by period, treatment and period effects may be exactly aliased. Diagnostics should therefore inspect the joint distribution of treatment, site, region, and period—not just the marginal site counts.

Worked Example: A Region With No Treatment-Group Overlap

Suppose a hypothetical historical control cohort comes only from the North region, where the observed outcome mean is 10 units. A current treatment cohort comes only from the South region, where the observed mean is 14 units. The pooled difference is \(14-10=4\) units. Can this be interpreted as a treatment effect after adjusting for region?

No. There are no treated observations in the North and no control observations in the South. The observed group difference combines treatment status with region. For illustration, write the mean as a treatment effect \(\tau\) plus a regional contrast \(d\), defined as the South-minus-North difference that would occur under control. The observed difference then satisfies \(4=\tau+d\). If \(d=0\), the implied treatment effect is 4 units; if \(d=3\), it is 1 unit; if \(d=5\), it is \(-1\) unit. The data described do not distinguish these possibilities.

A regression cannot estimate a data-supported region-adjusted treatment contrast when region perfectly predicts group. Any numerical answer would require an additional constraint, external information, or extrapolation assumption. A responsible report identifies the absence of overlap and avoids presenting the crude 4-unit difference as a region-adjusted treatment effect.

Diagnostics and Alternatives

Before fitting an adjusted model, inspect counts and outcome summaries by site, region, and period. Sparse cells can make estimates unstable even if there is nominal overlap. Compare baseline characteristics and outcome ascertainment across sites; check whether site labels and regional groupings are consistent over time; and identify sites that opened, closed, or changed their referral population. Where a site contributes only one group, flag it rather than allowing a model to obscure the lack of direct comparison.

The main analytical approaches have different targets and assumptions:

Approach	What it does	Main limitation
Crude pooled comparison	Compares all treated patients with all controls using their observed site mixes.	Can reflect geographic composition as well as treatment and time differences.
Site-stratified comparison	Compares groups within sites and combines site-specific contrasts.	Requires usable within-site comparisons; it does not by itself resolve temporal confounding.
Regression adjustment	Models outcome as a function of treatment, site, period, and selected covariates.	Depends on model specification and cannot solve exact lack of overlap or aliasing.
Standardization	Predicts or averages outcomes using a common, explicit target site distribution.	Requires support at target sites and correct estimation of the site-specific outcomes.

If site-specific treatment differences vary substantially, an additive model with one common treatment coefficient may be misleading. Report the pattern of site-specific estimates and consider whether a common-effect summary is scientifically meaningful. Conversely, large differences between site-specific estimates can reflect sampling variability, especially in small sites, rather than true effect modification. A hierarchical model can partially pool estimates, but partial pooling does not create information for wholly unrepresented treatment-site combinations and does not remove calendar-time confounding.

Common Pitfalls and Interpretation

Calling site adjustment a complete solution. Site terms can address measured site-level differences under the model, but they do not automatically control changing care, eligibility, or outcome measurement across time.
Reporting only the pooled contrast. A large difference may be driven by unequal site weights. Show site counts and site-specific outcome summaries alongside the pooled estimate.
Ignoring the target population. A standardized contrast depends on the chosen weights. State whether they represent the current patients, all eligible patients, or another target.
Extrapolating across empty cells without disclosure. If a target site or region has observations from only one group, an adjusted estimate relies on assumptions rather than a direct within-site comparison.
Interpreting between-site differences as treatment differences. Outcomes can differ across sites because of case mix or practice; those differences are not automatically caused by treatment.

A geographic comparability assessment is therefore part of the evidence for a historical control comparison, not a mechanical correction that guarantees validity. Its most useful result may be a finding that the available controls do not support the desired target comparison without strong assumptions.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

Why can a crude treatment-control difference differ from a weighted average of within-site differences?
Historical controls have a 15% outcome risk and current treated patients have a 19% risk. Both groups include Sites A and B, but their site proportions differ. What information is needed to calculate an equally weighted site-standardized risk difference?
Why does including both site and period indicators not necessarily separate geographic and calendar-time effects?
A region occurs only among historical controls. What does this imply for a region-adjusted treatment comparison in that region?

Solutions. (1) Each pooled mean uses that group's own site weights, so differences in site composition contribute to the crude contrast in addition to within-site differences. (2) The control and treatment outcome risks separately for Site A and Site B are needed; calculate each group's mean using weights 0.5 and 0.5, then subtract the standardized control risk from the standardized treatment risk. (3) Site and period may be strongly or perfectly associated, and treatment may be determined by period. With exact aliasing, the data cannot identify unrestricted separate effects for those variables. (4) There is no treated-group observation in that region to provide a direct within-region contrast; an estimate requires extrapolation or additional assumptions and should be reported as such.