Estimating Propensity Scores With External Controls

Why Estimate a Propensity Score?

An external-control comparison combines patients from different sources: trial participants and people observed in routine care or another data system. Even after eligibility criteria and time zero have been aligned, the groups may differ in their baseline characteristics. A propensity score summarizes, conditional on measured baseline covariates, the probability of belonging to a specified group. It can help describe how comparable the groups are and provide an input to later adjustment methods. It does not itself remove confounding.

The first decision is what probability the model should estimate. In many external-control analyses, the relevant score is the probability of trial participation rather than the probability of receiving a drug. These are not interchangeable. A trial may randomize treatment among its participants, while a nontrial control cohort is assembled from a separate source. The source of the comparison, the treatment definition, and the intended target population must therefore be stated before fitting a model.

The conditioning variables \(X\) must be available and meaningfully comparable in both sources. The earlier material on harmonizing baseline covariates is relevant here: two fields with similar names do not necessarily measure the same quantity, use the same time window, or encode the same categories. The earlier material on selecting confounders also informs which baseline variables belong in the model. A propensity model is not a device for selecting confounders automatically.

Define the Estimand’s Population Before Modeling

A participation score is only well defined relative to a population from which trial and nontrial patients could, conceptually, arise. Specify the eligibility rules, the time zero, the treatment strategies being compared, and how each source contributes patients. A score calculated in a convenience sample can still rank observed patients by their fitted probability of trial membership, but its numerical probabilities need not be the probabilities in the intended source population.

This issue arises when analysts deliberately sample a fixed number of trial participants and external controls. If the sampling fractions differ from the source-population fractions, the fitted model’s intercept generally reflects those sampling fractions. Its covariate associations and ranking may remain informative under appropriate sampling assumptions, but its probabilities should not be called calibrated source-population participation probabilities without correcting for sampling or using appropriate source-population information.

The model also requires a coherent baseline: covariates must be measured before trial participation or treatment assignment as defined for the analysis, not after outcomes or follow-up begin. If coding, measurement windows, or missing-data handling differ by source, the model can learn differences in data capture rather than clinically relevant differences between patients. These problems cannot be repaired simply by adding more predictors.

Logistic Regression as a Practical Model

A common specification is logistic regression. For covariates \(X_1,\ldots,X_p\), write the linear predictor as \(\eta(X)=\beta_0+\beta_1X_1+\cdots+\beta_pX_p\). Then the modeled probability of trial participation is the inverse-logit of that predictor. Continuous covariates may require nonlinear terms, and clinically plausible interactions may be included when justified. Categorical variables require suitable indicator coding and a stated reference category.

For a treatment score, replace \(e(X)\) with \(g(X)\), and model \(A\) rather than \(S\). In a randomized trial with known allocation probability, that randomization probability is known by design; it is not necessary to estimate it as though assignment were observational. A fitted participation score may still be relevant when comparing trial participants with external patients, but it answers a different question.

What the Logistic Fit Estimates

With observed indicators \(S_i\) and covariate vectors \(X_i\), a logistic model treats each \(S_i\) as Bernoulli with probability \(e(X_i)\). Under independent observations and the specified model, its likelihood is the product of the Bernoulli probabilities. The maximum-likelihood estimate chooses coefficients that maximize this likelihood. In a Bayesian fit, the same likelihood can be combined with a stated prior on the coefficients to obtain a posterior distribution; predicted scores can then be summarized across posterior draws rather than treated as known constants.

A short derivation explains the logistic form. A linear predictor can take any real value, whereas a probability must lie strictly between 0 and 1. The odds \(e(X)/(1-e(X))\) are positive; taking their logarithm maps them to the real line. Modeling that log-odds as a linear predictor and solving for the probability gives the inverse-logit formula above. For a single binary covariate, the fitted probabilities in its two groups can be calculated directly from the observed proportions, as in the next example.

Model Checks and Failure Modes

A fitted score is a model-based summary, so inspect the data and model before using it. Check that each covariate has the same definition and reference coding across sources, that covariates are genuinely baseline, and that all included terms can be estimated. Inspect fitted probabilities and their distributions separately by source. These checks can reveal coding errors, unsupported covariate patterns, or a model that nearly predicts source membership perfectly.

Complete or near separation occurs when a covariate pattern, or a combination of predictors, almost perfectly distinguishes trial participants from external controls. Ordinary maximum-likelihood logistic estimates can then become extremely large or fail to converge, and predicted probabilities can be close to 0 or 1. Sparse categories and an excessive number of interactions can create similar instability. Possible responses include checking coding and data provenance, simplifying unsupported terms, combining categories only when scientifically defensible, or using a regularized or Bayesian model with an explicitly justified prior. A prior can stabilize estimation, but does not create information where the sources have no comparable patients.

A score can also be numerically well behaved while the scientific comparison remains weak. Important prognostic or confounding covariates may be unmeasured; observed covariates may be measured with error; or the selected covariates may not capture meaningful differences in care setting or calendar time. Model fit and a plausible score distribution do not establish conditional exchangeability, meaning that the relevant potential outcomes are comparable across groups after conditioning on measured covariates. That assumption requires substantive justification and sensitivity analysis.

Comparing Approaches and Understanding Their Limits

Logistic regression is interpretable and widely available, but it imposes a linear form on the log-odds unless nonlinear terms and interactions are included. Flexible alternatives, such as splines or other prediction algorithms, can represent more complex patterns, but require careful tuning and validation and may be less transparent. No algorithm can compensate for incompatible variable definitions, omitted important covariates, or lack of information in one source. The choice should be guided by the scientific structure, data support, and stability of predictions, not by a single measure of predictive accuracy.

The participation score is not an outcome model: it predicts source membership or treatment assignment, not the clinical endpoint. It is also not an estimated treatment effect. Matching, weighting, and outcome-regression procedures use covariates or scores in different ways; their implementation and diagnostics are separate tasks. In particular, a score model that predicts group membership well is not automatically a good adjustment model. The causal purpose is to support a defensible comparison, not to maximize the ability to identify which source a patient came from.

For Bayesian analyses, coefficient uncertainty can be carried into score predictions. If \(\beta^{(m)}\) is a draw from the coefficient posterior, calculate \(e^{(m)}(X)\) for each draw. The posterior mean score is the average of these transformed draws, rather than necessarily the inverse-logit evaluated at the posterior mean coefficient, because the inverse-logit is nonlinear. This accounts for uncertainty in model coefficients, but it does not resolve uncertainty from unmeasured confounding or model misspecification.

Common Mistakes

• Using the wrong indicator. A model for trial participation does not estimate treatment receipt. State whether the modeled response is \(S\) or \(A\).
• Interpreting a score as an outcome probability. A participation score concerns source membership, not response, survival, or treatment benefit.
• Taking probabilities literally after unequal sampling. If trial and external patients were sampled at different rates, the fitted intercept may not represent source-population participation probabilities.
• Assuming a fitted score proves exchangeability. A model uses measured covariates; it cannot guarantee that all important confounding has been measured or correctly modeled.
• Treating a large model as automatically safer. Unsupported interactions and sparse categories can produce unstable estimates or separation. Model terms need data support and scientific justification.