Evaluating External-Control Selection Bias

Why Selection Into an External-Control Dataset Matters

An external-control analysis may begin with patients who could, in principle, have been eligible as controls, but include only those captured in a nontrial dataset with the records needed for analysis. These are two selection steps: a patient must enter the dataset, and the required information must be available. If either step favors patients with different prognoses, the observed controls may not represent the control population relevant to the trial. Adjustment for treatment-group differences does not, by itself, correct this separate selection problem.

The goal here is to evaluate those selection steps and determine whether weighting can make the observed controls more representative. This is not a guarantee that an external comparison is unbiased. It is an assessment that depends on a defined target population, measured covariates, and explicit assumptions about how selection relates to outcomes.

The target population must be stated before selection rates are interpreted. It might be all patients meeting trial eligibility criteria in a specified health system and period, rather than all patients in a registry or all patients treated in routine care. The denominator used to estimate selection probabilities must correspond to that target or to a clearly defined source population that can be related to it. A dataset containing only selected records does not, on its own, reveal how many eligible patients were never captured or lacked required records.

Selection summaries should separate the stages where possible. Report the number eligible for the source frame, the number included in the nontrial dataset, and the number with usable records. Within clinically important covariate groups, calculate both the dataset-inclusion rate and the record-availability rate among those included. Compare the covariate distribution of the target or source population with that of the final analysis sample. A selection rate can differ substantially across groups even when the selected sample looks large.

Selection Summaries and a Worked Diagnostic

For a binary covariate, a simple diagnostic is the difference in its proportion between the source population and the selected analysis sample. For several covariates, standardized differences can put differences on comparable scales. For a binary characteristic, one common standardized difference is

The sign indicates the direction of the difference; its magnitude indicates its size relative to the pooled standard deviation. It is a descriptive diagnostic, not a test that establishes selection bias. A covariate difference matters for outcome estimation when that covariate predicts the outcome or is related to other relevant factors.

When Inverse-Probability Selection Weighting Is Justified

If selection is adequately explained by measured baseline covariates, inverse-probability selection weighting can give greater influence to observed patients who resemble people less likely to be selected. For an observed patient with covariates \(X_i\), the basic weight is \(w_i=1/\pi(X_i)\). In practice, \(\pi(X)\) may be known from the sampling design or estimated from a model using information on selected and nonselected people. The model must represent the actual selection process, including required-record availability if the analysis sample requires those records.

The key identification assumption is conditional exchangeability of selection for the control outcome: within levels of the measured covariates \(X\), selected and nonselected patients have the same mean control outcome. In potential-outcome notation, this is \(Y^0\perp A\mid X\), where \(Y^0\) is the outcome under control. Weighting also requires positivity: every covariate pattern in the target population must have a nonzero probability of selection, \(\pi(X)>0\). The outcome and covariates must be defined and measured comparably, and the selection probabilities must be known or estimated adequately. Covariates used for adjustment should be measured before the outcome and should be relevant to selection and outcome representation.

Under these conditions, an inverse-probability-weighted estimate of the target mean control outcome is the weighted mean among observed controls. The justification follows by conditioning on \(X\):

The second equality uses conditional exchangeability, and the last uses \(\pi(X)=P(A=1\mid X)\). Averaging over the source-population covariate distribution yields the source-population mean; it yields the target mean only when the target and source populations have the same covariate distribution. With observed data, the normalized, or Hájek, version below estimates that source-population mean.

If the target and source covariate distributions differ, estimate the conditional control outcome mean among observed controls, \(\hat m_0(x)\), and standardize it over the target covariate distribution:

This requires the target covariate distribution to be known or estimable; conditional exchangeability identifies the target conditional mean from the observed controls.

With known probabilities, the corresponding unnormalized Horvitz–Thompson total uses \(\sum_{i:A_i=1}Y_i^0/\pi(X_i)\) and estimates the source-population total; dividing by a known source-population size estimates the source mean. This estimates the target mean when the target is the source population (or when their covariate distributions agree); otherwise, use target-standardization or another estimator that accounts for the target distribution. The normalized form above is often convenient when the target size is not known. Its finite-sample behavior and uncertainty still depend on the sampling and probability-estimation procedures.

Accounting for Record Availability

When the analysis requires complete records, model the probability of both steps rather than treating dataset inclusion as the only selection event. By the multiplication rule,

This expression does not assume that the two stages are independent. The second probability is conditional on having entered the dataset. If the stages are modeled separately, their estimated probabilities must be combined for each patient, and the inverse of that product is the weight for the final analysis sample.

Diagnostics, Alternatives, and Limitations

After weighting, compare the weighted covariate distribution with the intended target distribution. Large remaining differences indicate that the selection model or weighting procedure has not achieved its covariate-representation goal. Inspect the distribution of estimated selection probabilities and weights: probabilities near zero produce very large weights, making estimates unstable and highly sensitive to a few observations. Also check whether estimated weighted totals reproduce known source-population counts or covariate margins. These checks assess implementation and overlap; they cannot verify that all outcome-relevant selection factors were measured.

Weighting is one option, not a universal remedy. Matching can restrict the analysis to selected controls comparable on measured characteristics, at the cost of changing the represented population or discarding observations. Outcome regression can predict outcomes across covariate patterns, but then relies on a correctly specified outcome model. Doubly robust methods combine weighting and outcome modeling, while still requiring the relevant identification assumptions. These approaches address measured differences in different ways; none repairs unmeasured selection factors or a lack of support in the data.

Truncating extreme weights may reduce variance but changes the estimator and can reintroduce selection bias. Report the choice and examine whether conclusions depend on it. A further limitation arises if selection or record availability depends on the outcome even after conditioning on measured \(X\). Then \(Y^0\perp A\mid X\) fails, so weighting by \(\pi(X)\) alone does not identify the target outcome mean. This possibility requires explicit scrutiny rather than being treated as resolved by a good covariate balance table.

In a Bayesian analysis, estimated selection probabilities are uncertain quantities. Treating fitted weights as fixed can understate uncertainty in the final outcome or treatment-effect posterior. When selection-model uncertainty is material, it should be propagated—for example, by drawing selection-model parameters from their posterior, calculating weights for each draw, and carrying the resulting weighted estimates into the outcome analysis. The final interpretation remains conditional on the selection assumptions; a posterior distribution does not remove bias from an incorrect selection model.