Why Overlap Needs Explicit Reporting
A propensity-score analysis compares people who received different treatments using measured baseline covariates. The propensity score is the probability of receiving treatment given those covariates. If people with some covariate profiles almost always receive one treatment, there may be few comparable people receiving the other. An analysis can then rely heavily on extrapolation or on a small number of observations, even when its reported estimate looks precise.
A report should therefore show how the estimated scores are distributed in each treatment group, identify regions with sparse or absent support, and state what the investigators did about them. If units were trimmed or the cohort was restricted, report the rule and the units removed. The restriction also matters for interpretation: the resulting analysis no longer necessarily represents the entire original eligible cohort.
Positivity is an assumption about treatment probabilities in the target population, not a property that a histogram can prove. A score plot is a useful diagnostic of practical overlap, but estimated scores summarize the covariates into one number. Similar scores do not guarantee that the groups have similar covariate profiles in every important dimension. Conversely, a sparse tail in a finite sample does not by itself establish that the true treatment probability is zero there.
Overlap is considered alongside the other assumptions required for the causal question, including appropriate control of measured confounding. Reporting a plot does not establish those assumptions. The purpose is to make visible where the comparison is well supported by the observed data and where it may be fragile.
Displaying Score Distributions
Show the estimated propensity-score distributions separately for treated and comparison units. A histogram, density plot, or empirical cumulative distribution plot can be useful. Use the same horizontal score scale for both groups; when comparing histograms, also use comparable bin widths and vertical scaling. Label the axes, identify the groups, state the analysis sample used to calculate the scores, and make the score range from zero to one visible.
A plot should be accompanied by numerical summaries or a table when tails are important. Report group-specific sample sizes and, as appropriate, score ranges, selected quantiles, and counts in regions judged sparse. A statement such as “the distributions overlapped” is hard to evaluate without showing how much of each group lies in the common region. State whether an apparent empty tail contains no observations or merely few observations.
Choose bins or smoothing settings that reveal meaningful patterns without making random fluctuations look like sharp boundaries. If the conclusion depends on a cutoff, explain how it was selected. A figure of estimated scores should not be presented as if it directly displayed the full multivariate covariate space.
Worked Example: Reporting Sparse Score Regions
Suppose a hypothetical cohort includes 120 treated and 180 comparison units. The following table groups their estimated propensity scores into five bands.
| Estimated score band | Treated, n | Treated, % | Comparison, n | Comparison, % |
|---|---|---|---|---|
| Below 0.10 | 6 | 5.0% | 54 | 30.0% |
| 0.10 to below 0.30 | 24 | 20.0% | 54 | 30.0% |
| 0.30 to below 0.70 | 60 | 50.0% | 54 | 30.0% |
| 0.70 to below 0.90 | 24 | 20.0% | 18 | 10.0% |
| 0.90 to 1.00 | 6 | 5.0% | 0 | 0.0% |
| Total | 120 | 100.0% | 180 | 100.0% |
For example, the treated proportion below 0.10 is \(6/120=0.050\), or 5.0%; the comparison proportion is \(54/180=0.300\), or 30.0%. At scores of 0.90 or above, there are six treated units and no comparison units. The middle band contains 60 treated and 54 comparison units, so it provides a much denser region of observed representation from both groups than the upper tail.
A careful report would say that the distributions differ substantially in the lower tail and that no comparison units were observed in the highest score band. It would not say that positivity is proven to fail for every covariate profile with a score above 0.90: the table reports estimated scores and finite-sample counts, not the true conditional treatment probabilities. The plot and table would make the practical concern inspectable.
Reporting Trimming and the Resulting Population
Trimming means excluding units according to a stated rule, often one based on estimated propensity scores or on observed support in both groups. A report should make the rule reproducible: give the score model or score definition used, the cutoff or support rule, whether endpoints are included, and whether the rule was prespecified or chosen after inspecting the data. Also state the order in which the rule was applied if more than one restriction was used.
For each group, report the number eligible before restriction, the number removed by the rule, the number retained, and the removal percentage. The percentage is calculated within that group, not using a combined denominator:
Explain the target population in concrete terms. If the rule retains only units with estimated scores between two cutoffs, the analysis concerns the restricted cohort defined by that rule, rather than automatically representing every member of the original eligible cohort. Restriction can improve the empirical basis for comparison, but it changes who the results describe. It does not repair unmeasured confounding or guarantee adequate overlap across all covariates.
Worked Example: Applying a Prespecified Score Restriction
Consider a hypothetical cohort with 300 treated and 200 comparison units. Investigators prespecify retaining units with estimated scores from 0.05 through 0.95, inclusive. The score counts are as follows: among treated units, 18 have scores below 0.05, 276 lie within the interval, and 6 exceed 0.95. Among comparison units, 4 have scores below 0.05, 184 lie within the interval, and 12 exceed 0.95.
The treated removal count is \(18+6=24\), leaving \(300-24=276\). The treated removal percentage is \[ \frac{24}{300}\times100\%=8.0\%. \] The comparison removal count is \(4+12=16\), leaving \(200-16=184\), and its removal percentage is \[ \frac{16}{200}\times100\%=8.0\%. \] Overall, \(24+16=40\) of the original 500 units are removed, or \(40/500\times100\%=8.0\%\). The group-specific figures remain important: the same overall percentage could conceal very different amounts of removal by treatment group.
A reproducible report could state: “We retained units with estimated propensity scores from 0.05 through 0.95, inclusive. This excluded 24 of 300 treated units (8.0%) and 16 of 200 comparison units (8.0%), leaving 276 treated and 184 comparison units.” The resulting analysis concerns the original cohort members meeting that score rule. It should not be described without qualification as representing the full original cohort.
The score cutoff defines a restriction, not a universal boundary between valid and invalid observations. Its justification should connect to the observed support and analytic goal, and the report should make clear whether the cutoff was selected before or after examining the distributions.
Common-Support Rules and Their Limits
One transparent descriptive rule is to identify the intersection of the groups’ observed score ranges. If the treated range is \([a_1,b_1]\) and the comparison range is \([a_0,b_0]\), the intersection is \([\max(a_1,a_0),\min(b_1,b_0)]\), provided the lower endpoint does not exceed the upper endpoint. This summarizes the shared range of observed scores. It does not guarantee dense support throughout that interval: a range can overlap while most observations cluster in different parts of it.
Worked Example: Summarizing an Observed Common Range
In a hypothetical sample, the treated scores range from 0.08 to 0.91 and the comparison scores range from 0.14 to 0.83. The observed common range is \[ [\max(0.08,0.14),\min(0.91,0.83)]=[0.14,0.83]. \] Suppose 9 of 100 treated units have scores below 0.14, and 7 of 100 comparison units have scores above 0.83. Restricting to this common range therefore retains 91 treated and 93 comparison units. The removal percentages are \(9/100\times100\%=9.0\%\) and \(7/100\times100\%=7.0\%\), respectively.
The restriction removes treated units from the low-score tail and comparison units from the high-score tail. A report should identify those group-specific removals and describe the retained population as cohort members whose estimated scores fell within 0.14 to 0.83 under the stated rule. The numerical range alone is not enough: additional summaries or the overlaid distributions should show whether either group is sparse within that interval.
The observed-range rule is sensitive to extreme observations and may retain a region where one group is very sparse. A fixed cutoff, a range-based rule, and an analysis that keeps all units while using a different estimator are distinct choices; each has different consequences. Do not imply that a trimming rule is the only way to address practical overlap concerns. Depending on the design, investigators may reconsider the target population or use another analytic approach, but they should still report the observed support and limitations.
If no units are trimmed, report that fact rather than omitting the issue. State whether the observed distributions showed sparse tails, whether extreme scores were present, and how those features affect the credibility or stability of the intended comparison. If units are trimmed, distinguish this overlap restriction from other exclusions, such as eligibility restrictions or missing-data exclusions; the reasons and denominators should remain clear.
Justifying the Support Summary
The common-range calculation follows directly from the definition of an intersection. A score belongs to both observed ranges only if it is at least as large as each group’s minimum and no larger than either group’s maximum. Thus it must satisfy
which is equivalent to
This derivation justifies reporting the intersection as a descriptive summary of observed score ranges. It does not establish that all covariate profiles within the interval are represented in both groups. It is therefore a starting point for transparent reporting, not a complete test of positivity.
A practical reporting sequence is:
Use a common score axis and identify the treatment groups, sample, and plotting choices.
Give group-specific counts or proportions in the tails or other regions relevant to the comparison.
Give the rule, endpoints, rationale, timing, and number and percentage removed from each group.
Explain which cohort members remain and avoid implying that an estimate automatically applies to those excluded.
Common Mistakes and Limitations
- Calling a score plot proof of positivity. It shows estimated one-dimensional scores in the observed sample, not the true treatment probabilities for every covariate profile.
- Reporting only an overall removal count. Give denominators and removal percentages separately for treated and comparison units so unequal exclusions are visible.
- Using “common support” as if it meant dense support. Overlapping ranges can still contain regions with very few units from one group; inspect and report the distribution within the range.
- Leaving the cutoff unexplained. A threshold selected after viewing results can affect the represented population. State how and when the rule was chosen.
- Claiming that trimming restores causal validity. Restriction may reduce extrapolation concerns, but it cannot by itself establish exchangeability, correct model specification, or adequate support in every covariate dimension.
A useful report pairs a clear figure with enough numerical detail to reproduce the conclusion. It distinguishes observed data limitations from assumptions, and it states plainly when the analysis represents a restricted rather than original cohort.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why does a propensity-score plot not prove the positivity assumption?
- A rule removes 12 of 150 treated units and 10 of 250 comparison units. What are the group-specific removal percentages?
- The treated observed score range is 0.10 to 0.88, and the comparison range is 0.18 to 0.94. What is their observed common range?
- What should be reported if a study observes no comparison units in the highest score band but does not trim any units?
Solutions. (1) The plot shows estimated scores in a finite sample and compresses the covariates into one dimension; it cannot establish true treatment probabilities or representation for every covariate profile. (2) Treated: \(12/150\times100\%=8.0\%\). Comparison: \(10/250\times100\%=4.0\%\). (3) The intersection is \([\max(0.10,0.18),\min(0.88,0.94)]=[0.18,0.88]\). (4) Report the group-specific counts or percentages and the empty comparison tail, state that no units were trimmed, and describe the practical support concern and its implications without claiming that positivity is definitively violated.