Selecting Controls When Data Availability Is Outcome-Related

Why Data Availability Can Change the Comparator

An external-control analysis uses outcomes recorded outside the randomized trial to estimate what trial participants might have experienced under control. The records that are available may not be a neutral sample of the people who received control care. For example, patients with a documented event may be more likely to enter a registry, while patients who remain well may receive less follow-up. Conversely, a severe outcome may prevent later data capture. If availability depends on the outcome, the observed control group can have a distorted outcome distribution even when its measured covariates appear comparable with those of the trial.

Covariate adjustment and transport weights address differences in measured characteristics and their distributions. They do not, by themselves, correct selection that still depends on the outcome after conditioning on those characteristics. This tutorial focuses on making that remaining dependence explicit and assessing how conclusions change under plausible assumptions about it.

The target is usually the control outcome distribution in the trial population. A useful set of assumptions is that the recorded endpoint has the same definition and follow-up meaning in the trial and external source; that, conditional on \(X\), the control potential-outcome distribution is transportable from the source population to the trial population; and that there is adequate overlap in \(X\). These assumptions do not imply that \(R\) is independent of \(Y^0\) given \(X\). That independence is an additional condition, and the sensitivity analysis examines departures from it.

The distinction matters even if \(X\) includes strong measured predictors of the outcome. Selection may depend on the realized outcome, an unmeasured aspect of prognosis, or a process such as follow-up intensity that remains associated with the outcome after adjustment. No method can recover that dependence from selected records alone without additional information or assumptions.

A Binary-Outcome Selection Model

For a binary endpoint, define \(p(x)=P(Y^0=1\mid X=x)\) in the source population before selection, and \(p_s(x)=P(Y^0=1\mid X=x,R=1)\) among records that are available. The ratio of inclusion probabilities for people with and without the event is

A value \(\lambda(x)=1\) represents no outcome-related selection within covariate stratum \(x\). A value greater than 1 means event records are more likely to be available; a value less than 1 means they are less likely. This is a ratio of probabilities, not a logistic-regression odds ratio for inclusion.

The relation follows by applying Bayes’ rule to the event and non-event probabilities among records with \(R=1\). It does not require a rare outcome approximation.

Solving this equation for \(p(x)\) gives the correction above. To estimate the control risk for the trial population, correct within covariate strata and then average using the trial population’s covariate distribution. If the trial participants have covariate values \(X_i\), the corresponding standardization is

This order is important: when selection depends differently on the outcome in different covariate strata, correcting one overall risk and then adjusting for covariate composition need not give the same answer.

Standardizing Across Covariates

In practice, outcome-related availability may vary with measured prognosis. A single \(\lambda\) is then a simplifying sensitivity model, not a universal property of the dataset. Specify \(\lambda(x)\) by clinically meaningful strata or by a model that varies with selected covariates. Combine the corrected stratum-specific risks using the trial covariate distribution. This separates two issues: transporting outcomes across covariate distributions, and correcting outcome-dependent availability within covariate values.

Bayesian Uncertainty and Sensitivity Analysis

A sensitivity parameter is not automatically learned from the observed external outcomes. If only records with \(R=1\) are observed and the number or covariate profile of unavailable people is unknown, the selected records generally cannot distinguish a high source-population event rate with weak event selection from a lower source-population rate with stronger event selection. The analysis should therefore show results across defensible values of \(\lambda\), or assign an explicit prior to \(\lambda\) when a Bayesian probability statement about selection is justified by substantive information.

For each fixed \(\lambda\), uncertainty in the selected risk must still be propagated. In a Bayesian analysis, one can draw \(p_s(x)\) from its posterior distribution, transform every draw using the correction formula, and standardize each corrected draw to the trial covariate distribution. This produces a posterior distribution for the adjusted control risk conditional on the chosen \(\lambda\). If \(\lambda\) is assigned a prior, draw it as well and report that the posterior then depends on that prior and on the selection model. A credible interval conditional on a fixed \(\lambda\) does not include uncertainty about whether that \(\lambda\) is plausible.

Continuous Outcomes and an Exponential Tilt

For a continuous outcome, an analogous sensitivity model specifies how selection changes the outcome density. One useful model is an exponential tilt: within covariate stratum \(x\), the density among available records is proportional to \(e^{\delta y}\) times the source-population density. Here \(\delta\) measures selection by outcome and has units inverse to the outcome’s units. A positive value tilts the selected distribution toward larger outcomes. This model is a sensitivity assumption; it is not identified merely by observing the selected density.

Diagnostics, Alternatives, and Limits

There is generally no definitive diagnostic for outcome-related selection using only the selected records: the missing outcomes are precisely what would reveal the selection pattern. Useful evidence may come from a known source-population denominator, linkage to another data source, validation samples with more complete outcome ascertainment, or recorded indicators of follow-up and inclusion. Where inclusion can be compared across groups with known outcomes, examine whether availability differs by outcome after accounting for measured covariates. Such evidence can help calibrate sensitivity values, but it does not establish that unobserved selection is absent.

A practical analysis can report a range of \(\lambda\) values, explain their clinical interpretation, and show whether the treatment comparison changes materially across that range. Include values on both sides of 1 when both directions are credible. Check that the assumed selection probabilities could be consistent with any known inclusion rates, and assess whether there is covariate overlap and sufficient data in each stratum. Sparse strata make the selected-risk estimates unstable; near-zero or near-one selected risks can also make corrections highly dependent on the selection assumption.

The main alternatives address different problems. Transport weighting rebalances measured covariates to the target population but does not remove outcome-dependent selection within covariate strata. Outcome regression can model control risk conditional on covariates, but remains vulnerable if the available records are selected on outcome beyond those covariates. If a validation source supplies information about unavailable outcomes, a selection model may be estimated or better calibrated; without such information, the dependence must be treated as a sensitivity assumption. These approaches can be combined, but their assumptions should be stated separately.

A common mistake is to say, “The external controls were weighted to match trial covariates, so their outcomes are representative.” Matching measured covariate distributions does not imply that available records represent all source patients with those covariates. The correction is to ask whether availability is independent of the control outcome conditional on the adjustment covariates. If that condition is not credible, show how results vary under explicit outcome-selection assumptions. A second limitation is that this analysis addresses selection into the available dataset; it does not repair endpoint misclassification, unmeasured confounding, or noncomparability of care and follow-up.