Diagnosing Positivity and Covariate Overlap

Why Overlap Is a Design Question

An external control group can be large and still provide little information about some trial participants. If patients in an important clinical subgroup are common in the trial but rare—or absent—in the external cohort, an adjusted comparison for those patients must rely heavily on extrapolation. This is a problem of positivity and covariate overlap, not something that can be diagnosed by looking only at the total sample size.

Here let \(A=1\) identify participants in the trial treatment group and \(A=0\) identify external controls. Let \(X\) denote measured baseline covariates relevant to treatment assignment and outcome, and define the propensity score \(e(X)=P(A=1\mid X)\). The score summarizes how likely a covariate profile is to occur in the trial group rather than the external-control group. It does not measure the probability that a patient benefits from treatment.

The condition is target-specific. If the goal is an effect for the trial population, profiles found only among external controls do not by themselves prevent estimation for that target. The key question is whether trial participants have adequate comparator support. A broader target that includes both sources would require support in both directions.

Positivity is considered alongside conditional exchangeability (no residual confounding given the measured covariates), consistency, and adequate measurement of relevant covariates. Overlap cannot establish these assumptions. It is a necessary support condition for the intended adjusted comparison, not proof that the comparison is unbiased.

What Propensity-Score Overlap Means

If both groups occur at a covariate profile, then \(0<e(x)<1\). Scores close to one indicate profiles much more common in the trial group than among external controls; scores close to zero indicate the reverse. A score near one is concerning when the target includes those trial participants, because there may be few comparable controls.

The relationship between the score and the covariate distributions follows directly from Bayes’ rule. Write \(f_1(x)\) and \(f_0(x)\) for the covariate densities among trial participants and external controls, and let \(\pi=P(A=1)\). For profiles where the densities are positive,

Thus, if a profile occurs among trial participants but not in the external-control population, \(f_0(x)=0\), the score is one there. When \(f_0(x)\) is positive but very small relative to \(f_1(x)\), the score can be close to one, signalling weak practical support. In finite data, an estimated score of exactly zero or one may also reflect model separation or sparse observations; it is a warning to investigate, not automatic proof of a population-level structural zero.

Inspecting Scores and Covariate Support

A useful diagnosis combines summaries of estimated propensity scores with direct inspection of the covariates that define clinical comparability. Display the score distributions separately by group, using overlapping histograms or density plots, and inspect empirical ranges and quantiles. Look for trial observations in a tail with few or no external controls, and for large gaps between the groups. A single mean score for each group can hide such gaps.

Score plots are a summary, not a complete test. A fitted score model can compress very different covariate patterns into similar scores, especially if important interactions or nonlinear relationships are omitted. Conversely, a model can produce extreme scores because it is poorly specified or because a few sparse patterns nearly separate the groups. Inspect clinically important covariates and combinations of covariates directly—for example, disease severity jointly with biomarker status, rather than each marginal distribution alone.

This is a design-stage diagnostic. It asks whether the observed covariates provide a credible basis for comparing the groups in the chosen target. It is distinct from assessing covariate balance after an adjustment procedure. A score plot can reveal lack of support before adjustment, but it does not show that adjustment has achieved balance.

From Diagnosis to a Target-Population Decision

When overlap is weak, there is no purely statistical repair that recovers information absent from the data. The response should make the target population explicit. Depending on the clinical objective, investigators may retain the full trial target and acknowledge reliance on extrapolation, define a narrower target with adequate comparator support, or use a method that emphasizes profiles represented in both groups. The choice changes the population to which the result applies.

Trimming excludes observations outside a specified score or covariate-support region. If the original target is all trial participants, trimming changes the estimand to an effect for the retained trial participants, unless an additional, explicit standardization step defines another target. The retained population should be described by its clinical characteristics, not just by a score threshold. Report how many participants are removed and why the excluded profiles lack adequate support.

A threshold such as 0.10 or 0.90 is not a law of positivity. A useful cutoff depends on the design, the target, the score model, the severity of extrapolation, and clinical judgment. Investigators can examine how conclusions about the target’s composition and available support change across reasonable thresholds. That is a sensitivity check on the target definition; it does not make unsupported profiles supported.

Alternatives, Failure Modes, and Limits

The main alternatives express different compromises. Keeping the full target preserves the original clinical question but requires extrapolation for weakly supported profiles; conclusions may then depend strongly on model form. Restricting or trimming the target improves comparability by excluding unsupported profiles, but sacrifices generalizability to those excluded patients. Overlap weighting, introduced in the earlier material on overlap weighting, emphasizes patients in covariate regions shared by the groups; its target differs from the full trial population. These approaches are not interchangeable ways to produce the same estimand.

Weight truncation is not the same as fixing positivity. Limiting extreme weights can reduce numerical instability, but it does not supply actual controls for a trial profile that has no support. A more flexible propensity or outcome model may represent complex relationships better, but flexibility cannot replace missing comparator information. Bayesian estimation likewise does not remove this design limitation: posterior uncertainty may reflect assumptions and prior information, but the observed data still do not identify a comparison in a region with no support.

Practical warnings include very few controls in a clinically important trial subgroup, fitted scores concentrated near one among trial participants, disjoint ranges, and joint covariate cells present in the trial but absent in controls. Investigate whether warnings persist under reasonable score specifications and whether they correspond to clinically meaningful profiles. The propensity model should include relevant baseline covariates and plausible nonlinear terms or interactions; an apparent extreme score from a misspecified model is not automatically a true support failure. At the same time, changing the model solely to make the score distributions look better is not a valid substitute for evidence of comparability.