Why Overlap Matters
Inverse-probability weighting constructs a comparison by giving more influence to observations whose treatment status was relatively unlikely given their covariates. This can be useful when treatment assignment differs across covariate profiles, but it also creates a vulnerability: if a profile is very unlikely to occur in one treatment group, the few observations from that group can receive very large weights. If the profile never occurs in that group, the data do not supply the comparison needed there.
Overlap diagnostics examine whether treated and untreated observations occupy sufficiently similar covariate regions for the intended causal comparison. They do not establish that the groups are comparable on measured covariates, and they cannot verify that all confounders have been measured. Those are separate questions. The focus here is the availability of treatment and control observations across the target population.
Formal Setup: Positivity and Overlap
Let \(A\in\{0,1\}\) denote treatment, \(X\) the pre-treatment covariates, and \(Y(1)\) and \(Y(0)\) the potential outcomes under treatment and control. Define the propensity score \(e(X)=P(A=1\mid X)\). For the average treatment effect (ATE), the target population is the population represented by the covariate distribution of the study population.
The usual identification argument for the ATE relies, among other conditions, on consistency, conditional exchangeability \( (Y(1),Y(0))\perp A\mid X\), and positivity. Consistency links observed outcomes to the potential outcome under the treatment actually received. Conditional exchangeability rules out unmeasured confounding after conditioning on \(X\). The condition examined here is positivity: each covariate profile represented in the target population must have a positive probability of receiving either treatment.
The formal condition is about the population treatment probabilities, not merely about the observed sample. It does not require every treated and untreated observation to have the same covariate values. It does require that the target population not include profiles for which treatment or control is impossible. When probabilities are positive but very close to zero or one, the formal condition may hold while estimation remains unstable: this is often called a practical positivity problem.
Three Complementary Diagnostics
No single summary diagnoses overlap completely. A useful assessment combines propensity-score distributions, the resulting inverse-probability weights, and the effective sample size implied by those weights.
First, compare the distributions of estimated propensity scores separately for treated and untreated observations. A histogram, density plot, empirical cumulative distribution, or quantile table can reveal regions where one group has few or no observations. The propensity score is a one-dimensional summary of covariates, so its distributions are a convenient diagnostic, not a complete description of every covariate pattern. In particular, estimated scores depend on the fitted propensity model; misspecification can make their apparent overlap misleading.
For ATE inverse-probability weighting, the individual weights are
Thus a treated observation with an estimated score near zero, or a control observation with a score near one, receives a large weight. Such weights indicate that the observation represents a covariate region sparsely represented in its own treatment group. Report more than the maximum when possible: useful summaries include upper percentiles, the largest weights, and the fraction of total weight carried by a small number of observations. There is no universal cutoff that proves overlap is adequate or inadequate; the assessment depends on the target, sample size, and consequences of unstable weighting.
A third diagnostic is the effective sample size (ESS), calculated separately for each treatment arm using that arm's weights:
This summary expresses how concentrated the weights are in terms of an equivalent number of equally weighted observations. It does not literally change the sample size, and it is not a substitute for a variance estimate. In particular, it cannot capture every consequence of outcome variability or dependence between observations.
Worked Examples
Worked Example: Reading Propensity-Score Distributions
Suppose a hypothetical study estimates propensity scores for 10 treated and 10 control observations. The table groups the estimated scores into five intervals.
| Estimated-score interval | Treated count | Control count |
|---|---|---|
| [0, 0.1) | 0 | 3 |
| [0.1, 0.3) | 1 | 3 |
| [0.3, 0.7) | 4 | 3 |
| [0.7, 0.9) | 3 | 1 |
| [0.9, 1] | 2 | 0 |
The middle interval has observations from both groups, while the lowest interval contains only controls and the highest only treated observations. This is an empirical warning: estimates for the full target population may rely heavily on extrapolation or on a few observations near the boundaries. The counts do not prove that the population probabilities are exactly zero in either tail. Nor do they show whether the intervals contain similar covariate profiles; the table summarizes estimated scores, not all of \(X\).
Worked Example: Extreme ATE Weights and Effective Sample Size
Consider a hypothetical sample with three treated and three control observations. Their estimated propensity scores are, respectively, \(0.20, 0.80, 0.95\) for treated observations and \(0.10, 0.90, 0.05\) for controls. For treated observations use \(1/\hat e(X_i)\); for controls use \(1/\{1-\hat e(X_i)\}\).
The treated weights are
Their sum is approximately \(7.303\), and their sum of squares is approximately \(27.671\). Therefore,
Although there are three treated observations, their weights are as concentrated as three equally weighted observations would be only if there were about 1.93 such observations. The treated observation with score \(0.20\) carries most of the weight.
The control weights are
Their sum is approximately \(12.164\), and their sum of squares is approximately \(102.343\), giving
The control observation with score \(0.90\) receives a weight of 10 because controls are relatively rare at that score. Both arms show weight concentration, especially the control arm. These figures warn that the weighted comparison may be sensitive to a few observations; they do not, by themselves, quantify uncertainty or establish whether the causal assumptions hold.
Worked Example: A Sample Gap Is Not a Population Proof
Suppose a hypothetical covariate subgroup has a true treatment probability \(e(X)=0.02\), so positivity holds for that subgroup: treatment is rare, but not impossible. If 50 independent individuals from the subgroup are observed, the probability of seeing no treated individual is
Thus a sample can contain no treated observations in a subgroup even when the population treatment probability is positive. Conversely, observing at least one treated and one control observation in every displayed score interval does not prove that treatment probabilities are bounded away from zero and one throughout the target population. Finite data provide evidence about the design and fitted model, not direct proof of population probabilities at every covariate value.
Why the ESS Formula Reflects Weight Concentration
The ESS has a simple justification. Let \(q_i=w_i/\sum_j w_j\) be the normalized share of total weight carried by observation \(i\), so that \(\sum_i q_i=1\). Then
If all \(n\) observations have equal weights, each share is \(1/n\), and \(ESS=1/\{n(1/n^2)\}=n\). If one observation carries nearly all the weight, its share is close to one, the sum of squared shares is close to one, and the ESS is close to one. More generally, Cauchy–Schwarz gives \(1=(\sum_i q_i)^2\leq n\sum_i q_i^2\), so \(ESS\leq n\). This explains why concentrated weights reduce ESS. It is an effective-count interpretation, not an exact claim that the weighted estimator has the same variance as a simple random sample of that size.
Interpreting Diagnostics Carefully
Propensity-score plots, weight summaries, and ESS answer related but distinct questions. Score distributions reveal where treated and control observations appear sparse; extreme weights show how strongly the weighting procedure amplifies those sparse observations; ESS summarizes the resulting concentration within an arm. Concordant warnings across these diagnostics make instability more plausible than any one statistic alone.
These checks do not establish conditional exchangeability, which concerns whether relevant confounders have been measured and adjusted for. They also do not establish covariate balance after weighting. Overlap and balance are different: overlap concerns whether the data contain observations capable of representing the required treatment comparisons, while balance concerns the covariate distributions after adjustment.
Common Mistakes
- Equating sample gaps with a proven positivity violation. A finite sample can omit a treatment group in a region even when its population probability is positive. Describe the observed gap as a warning and consider the design and target population.
- Calling positivity adequate because the formal condition is technically satisfied. Probabilities near zero or one can produce highly variable weights. Practical instability matters even when probabilities are not exactly zero or one.
- Reporting only the largest weight. One maximum can be an isolated feature, while many moderately large weights can also concentrate an arm's total weight. Include percentiles or weight shares and ESS where appropriate.
- Interpreting ESS as the actual sample size or a complete precision measure. ESS summarizes weight concentration; it does not replace appropriate inference or account for every feature of the outcome data.
- Assuming overlap diagnostics prove no unmeasured confounding. They assess support, not conditional exchangeability. A well-behaved score distribution cannot rule out unmeasured confounding.
Check Your Understanding
Try these questions using the definitions and calculations above.
- State the positivity condition for an ATE with binary treatment in terms of \(e(X)\).
- Why can a control observation with an estimated propensity score near one receive a large ATE weight?
- What does an arm-specific ESS of 2.4 suggest if that arm contains 20 observed individuals? What does it not prove?
- Explain why a score interval containing only treated observations is an empirical warning but not, by itself, proof of a population positivity violation.
- Name one causal assumption that propensity-score overlap diagnostics do not verify, and briefly state what it concerns.