Why Poor Overlap Changes the Question
Matching compares treated and control units with sufficiently similar measured covariates. When some treated units have no comparable controls, or some controls have no comparable treated units, a matching procedure may exclude them. This can improve the plausibility of comparisons among the units that remain, but it also changes which population an estimate represents. The central task is therefore not just to calculate a difference: it is to identify who remains eligible for comparison and define the effect for that population.
Let \(A\) indicate treatment, with \(A=1\) for treated and \(A=0\) for control, let \(X\) denote measured pretreatment covariates, and let \(Y(1)\) and \(Y(0)\) be the potential outcomes under treatment and control. Write \(e(X)=P(A=1\mid X)\) for the propensity score. Let \(S=1\) indicate that a unit belongs to the population retained by a specified support restriction or matching procedure. The restriction should be described in terms of the covariate profiles or matching eligibility that determine \(S\), not merely by saying that “the data were matched.”
A restriction can produce a target that differs from both the full-population ATE and the original ATT. Which target is appropriate depends on the actual retained units and the matching weights. In particular, a matched estimate based on retaining treated units and finding controls for them often targets the retained treated population. It should not automatically be labelled the restricted-population ATE.
Conditions for a Causal Interpretation
Restricting to better-overlapped units does not, by itself, make treatment assignment unconfounded. A causal interpretation still needs substantive assumptions. Consistency means that the observed outcome is \(Y=Y(A)\). Conditional exchangeability requires \((Y(1),Y(0))\perp A\mid X\): after conditioning on the measured covariates, treatment assignment is independent of both potential outcomes. Positivity requires both treatment states to be possible at the covariate values in the target population.
Under these assumptions, the restricted ATT can be written as a standardization over the covariate distribution of retained treated units:
The equality follows because consistency gives \(E(Y\mid A=a,X)=E[Y(a)\mid A=a,X]\), while conditional exchangeability lets us replace the conditional expectation among units with \(A=a\) by \(E[Y(a)\mid X]\). Averaging those conditional differences over the covariate distribution of retained treated units gives the restricted ATT. If the matched controls do not adequately represent the control outcomes at those covariate values, the equality does not rescue the estimate: the matching comparison has failed to approximate the required standardization.
Worked Example: Matching When One Profile Has No Controls
Worked Example: Matching When One Profile Has No Controls
Consider a hypothetical dataset with three exact covariate profiles. A matching rule retains treated units only when a control with the same profile is available. The outcome is measured in points. The following table reports the counts and observed mean outcomes.
| Profile | Treated count | Control count | Treated mean outcome | Control mean outcome |
|---|---|---|---|---|
| A | 4 | 6 | 12 | 10 |
| B | 2 | 4 | 18 | 10 |
| C | 4 | 0 | 25 | Not observed |
Step 1: Identify the overlap problem. Profiles A and B contain both treated and control units. Profile C has four treated units and no controls, so the treatment effect for C cannot be estimated by a within-profile comparison. The stated matching rule therefore excludes the four profile-C treated units.
Step 2: Calculate the profile-specific differences. For A, the observed difference is \(12-10=2\) points. For B, it is \(18-10=8\) points. Under the assumptions above, and treating the displayed means as the relevant within-profile outcome means, these differences estimate the effects for their respective profiles.
Step 3: Calculate the matched treated-target estimate. All six retained treated units are represented. With one matched control per treated unit, four pairs come from A and two from B:
This is an estimate for the retained treated units: four treated units in A and two in B. It is not an estimate for the four excluded treated units in C.
Step 4: Compare with the ATE for everyone in the restricted population. If the target instead includes all units in profiles A and B, the profile sizes are 10 in A and 6 in B. The corresponding standardized estimate is
The two estimates differ because the restricted treated population weights the profiles by treated counts (4 and 2), whereas the restricted-population ATE weights them by total population counts (10 and 6). Neither calculation supplies an effect for profile C. The full-population ATE remains unresolved without additional information or assumptions about the effect for C.
Reading the Retained Population From the Data
A practical overlap assessment starts by describing the covariate distributions in the treatment groups and locating regions where both groups are represented. Propensity-score summaries can help, but a score plot alone does not establish that all important covariates are balanced. Inspect the covariates used to form the score as well, and examine whether the retained controls actually resemble the retained treated units on those covariates.
The next example illustrates how a score restriction changes the treated population. The score bands and counts below are hypothetical; the average scores summarize fitted propensity scores within each band.
Worked Example: Quantifying Who Remains After a Score Restriction
Suppose a dataset has three bands of fitted propensity scores. A prespecified restriction retains units in the low and middle bands but excludes the high band, where very few controls are available.
| Band | Average fitted score | Treated count | Control count | Illustrative effect (points) |
|---|---|---|---|---|
| Low | 0.15 | 3 | 17 | 1 |
| Middle | 0.55 | 11 | 9 | 5 |
| High | 0.95 | 19 | 1 | Not used |
There are \(3+11+19=33\) treated units in total, and the restriction retains \(3+11=14\). Thus the retained share of treated units is
or about 42.4%. The retained treated population consists of three low-band and eleven middle-band units; it does not represent the nineteen high-band treated units. If the illustrative band effects are valid estimates for their respective treated subgroups, the restricted ATT is
For comparison, the restricted population includes 20 units in each retained band: \(3+17=20\) in the low band and \(11+9=20\) in the middle band. Its ATE, using these band-specific effects, is
The different values are not a calculation error. The restricted ATT gives greater weight to the middle band because most retained treated units are in that band. The restricted ATE gives equal weight to the two bands because their total retained population sizes are equal. The high-score treated units are outside both targets under this restriction.
These calculations also show why reporting only a matched effect is incomplete. A reader needs to know how many treated units were excluded, which covariate regions were excluded, and whether the stated target is the retained treated population or all units in the retained region.
Why Broad Profiles Can Hide a Target Shift
A profile may combine covariate subtypes that have different treatment effects and different treatment probabilities. In that case, a profile-wide ATE is not necessarily the effect among treated units in that profile. The appropriate weights depend on the target: average subgroup effects over all units for an ATE, and over treated units for an ATT. This distinction matters even when treatment is exchangeable after conditioning on the finer covariates.
Worked Example: Heterogeneous Effects Within a Profile
Suppose one broad profile contains 20 hypothetical units: 10 of subtype U and 10 of subtype V. One unit in U and nine units in V are treated; the remaining nine U and one V units are controls. Assume treatment is exchangeable conditional on subtype, both treatment states occur in each subtype, and the effects are 0 points for U and 10 points for V.
The profile-wide ATE weights the two equally sized subtypes equally:
The effect among treated units in the profile instead weights by the treated subtype composition. One treated unit is in U and nine are in V, so
Thus, weighting the profile-wide ATE of 5 points by the treated count would not recover the treated-target effect of 9 points. If the matching analysis uses only the broad profile and fails to balance subtype, its comparison may not identify either target. A valid treated-target standardization must preserve or account for the subtype distribution among treated units.
In general, if \(k\) indexes exact covariate profiles, define \(\tau_k\) as the average treatment effect among retained treated units in profile \(k\) when calculating a restricted ATT. Then weighting by the retained treated counts \(n_{1k}\) is appropriate:
For the restricted-population ATE, define \(\tau_k\) instead as the whole-profile average effect and weight by the profile population counts \(N_k\):
These formulas use different profile-specific definitions of \(\tau_k\). If profiles combine effect-modifying subtypes, the two definitions need not coincide. With exact covariate profiles, or with constant effects within each profile, the distinction disappears.
Diagnostics, Alternatives, and Limitations
Several checks help reveal when a poor-overlap analysis is going wrong:
- Count exclusions by treatment group and covariate region. A small overall exclusion percentage can conceal the loss of a clinically or substantively important subgroup.
- Inspect score and covariate distributions before and after restriction. If the retained groups remain markedly different, matching has not created adequate comparability.
- Check whether controls are reused or heavily weighted. A result driven by a few controls may be unstable and may represent a narrower set of comparisons than the nominal sample size suggests.
- Look for residual heterogeneity within matching profiles. Broad categories can hide effect modifiers or imbalance in important covariates.
- State the target in words and with a count. For example: “the average effect among the 14 treated units retained in the low and middle score bands,” not simply “the treatment effect.”
Restriction is not the only response to poor overlap. Analysts may choose a different matching specification, use weighting that emphasizes regions of overlap, or model outcomes and standardize to a defined population. These alternatives answer different questions and rely on different modeling or weighting choices; they do not automatically recover effects for regions with no support. In particular, extrapolating outcomes for a treatment group absent from a covariate region depends on assumptions that the observed data cannot verify there.
A restricted estimate can be useful when the retained population is scientifically relevant and the comparisons within it are credible. Its main limitation is scope: it may not describe excluded units, and it does not by itself justify transporting an effect to the full population. Another limitation is that estimated overlap and matching eligibility may depend on the observed sample and chosen procedure. The target should therefore be reported alongside the effect estimate, rather than treated as a technical detail.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A matching rule retains 8 of 12 treated units and excludes 4 treated units with no comparable controls. Which treated population does a matched treated-target estimate describe?
- In a restricted sample, profile A has 6 treated and 4 control units, with an effect of 2 points. Profile B has 2 treated and 8 control units, with an effect of 7 points. Calculate the restricted ATT and restricted-population ATE.
- Why does conditional exchangeability alone not allow an estimate for a covariate profile containing treated units but no controls?
- A broad profile has two equally sized subtypes with effects 2 and 8 points. Treated units are 25% from the first subtype and 75% from the second. Calculate the profile ATE and the effect among treated units.
Solutions. (1) It describes the 8 retained treated units, not all 12 treated units. (2) The restricted ATT is \([6(2)+2(7)]/(6+2)=26/8=3.25\) points. The restricted ATE is \([10(2)+10(7)]/20=4.5\) points because the profile population counts are 10 and 10. (3) With no controls in that profile, the control potential-outcome mean is not observed there; exchangeability cannot supply the missing empirical comparison or establish positivity. (4) The profile ATE is \((2+8)/2=5\) points. The treated-target effect is \(0.25(2)+0.75(8)=6.5\) points, because it weights effects by the treated subtype distribution.