Why Reporting Is Part of the Design
A positivity or overlap diagnostic is useful only if readers can see what was examined and what decisions followed. A statement such as “poor overlap was addressed by trimming” leaves key questions unanswered: which score ranges were unsupported, which observations were removed, how many were removed from each treatment group, and what population remains represented? These are not cosmetic details. A restriction can change the population to which an effect estimate pertains.
This tutorial focuses on reporting those details clearly. It does not prescribe a universal trimming threshold: no single cutoff is appropriate for every study. The report should make the evidence, rule, and consequences inspectable, including when no restriction is made.
A good report separates three things: what the observed scores look like, what rule was applied, and what population the resulting analysis represents. In particular, ranges alone can conceal sparse tails, and an overall score summary can conceal a subgroup with little or no support. Report enough information for a reader to understand the diagnosis without implying that a sample-based diagnostic proves positivity.
What to Report About the Score Distributions
Describe the score model and the sample to which its scores refer. At minimum, identify the covariates used to estimate the scores, the number of treated and control units with scores available, and how missing or unusable scores were handled. When scores are estimated in a particular analysis sample, say so. A score distribution is conditional on that modeling and sample choice; it is not a fixed feature of the study independent of those decisions.
Present summaries separately for treated and control units. Useful summaries include the minimum, selected quantiles (often the 10th, 25th, 50th, 75th, and 90th percentiles), and maximum. A plot of the two distributions can reveal concentration and tail differences that a range or a few quantiles miss. State the scale and give the sample size for each group. If a histogram or density plot is used, specify that it displays estimated scores by treatment group and use comparable axes and bins or smoothing choices.
Overall summaries are not enough when support may differ across important covariate subgroups. Report subgroup summaries or a diagnostic table for substantively relevant strata, such as age bands or disease-severity categories. Identify how strata were defined and give group-specific counts. A subgroup may have no observed controls, for example, even if the overall score distributions appear to overlap. Conversely, small sample counts can make a subgroup diagnostic unstable; report those counts rather than presenting an apparently precise conclusion.
Worked Example: Summarizing Scores by Treatment Group
Suppose a hypothetical study has estimated scores for 100 treated and 120 control units. Its score summaries are:
| Group | n | Minimum | 10th percentile | Median | 90th percentile | Maximum |
|---|---|---|---|---|---|---|
| Treated | 100 | 0.08 | 0.24 | 0.68 | 0.94 | 0.98 |
| Control | 120 | 0.02 | 0.07 | 0.31 | 0.76 | 0.90 |
The treatment groups have different score distributions: the treated median is 0.68 compared with 0.31 among controls, and the control 10th percentile is 0.07 compared with 0.24 among treated units. The ranges overlap from 0.08 to 0.90, but that fact alone does not show that support is adequate throughout that interval; the density and subgroup distributions still matter.
A transparent report might say: “Among 100 treated and 120 control units with estimated scores, the treated median score was 0.68 (10th–90th percentiles, 0.24–0.94; range, 0.08–0.98), and the control median was 0.31 (10th–90th percentiles, 0.07–0.76; range, 0.02–0.90). The distributions differed, particularly in their lower and upper tails; we also examined the distributions within prespecified covariate subgroups.” The last clause should be followed by actual subgroup findings if those analyses were conducted.
State the Restriction Rule Precisely
If trimming or another support restriction is used, report the exact rule in terms that can be applied to an individual observation. Include the threshold values, whether the bounds are inclusive, whether the same bounds apply to both groups, and whether restrictions are based on estimated scores, observed ranges, or a different prespecified criterion. For instance, the rule \(0.10 \leq \hat e_i \leq 0.90\) retains observations on both boundaries and excludes scores strictly below 0.10 or strictly above 0.90.
Do not report only “we trimmed extreme scores.” That phrase does not identify which observations were excluded. If a rule uses treatment-specific support, state the criterion separately for each group; a rule that restricts treated units to control support is not interchangeable with symmetric trimming. If thresholds were selected after inspecting score distributions, describe how. Report any prespecified thresholds distinctly from sensitivity analyses using alternative thresholds.
The number of units excluded should be shown separately by treatment group, with both counts and denominators. Give retained counts too, so readers can check the arithmetic. If units were excluded for other reasons, such as missing covariates, distinguish those exclusions from trimming; otherwise the reported trim proportion is ambiguous. If group-specific exclusion proportions are useful, compute them as excluded divided by the original score-analysis sample in that group.
Worked Example: Counts Under a Symmetric Threshold
Continue with 100 treated and 120 control units. A hypothetical analysis applies the symmetric rule \(0.10 \leq \hat e_i \leq 0.90\). Among treated units, 2 have scores below 0.10 and 18 have scores above 0.90. Among controls, 12 have scores below 0.10 and 3 have scores above 0.90. Assume these categories do not overlap.
For treated units, the number excluded is \(2+18=20\), so the number retained is \(100-20=80\). The excluded fraction is \(20/100=0.20\), or 20%. For controls, the number excluded is \(12+3=15\), leaving \(120-15=105\); the excluded fraction is \(15/120=0.125\), or 12.5%.
A concise report would state: “Using the inclusive estimated-score restriction \(0.10 \leq \hat e_i \leq 0.90\), we excluded 20 of 100 treated units (20.0%; 2 below and 18 above the bounds) and 15 of 120 controls (12.5%; 12 below and 3 above). The restricted sample contained 80 treated units and 105 controls.” The group-specific breakdown makes the direction of the restriction visible; a single total of 35 exclusions would not.
Describe the Population After Restriction
A restriction changes the set of units represented by the analysis. State this directly, rather than describing the retained sample merely as “the study population.” For a common score interval, a useful description is the population of eligible units whose true propensity scores lie in that interval, with the qualification that the implemented rule uses estimated scores. In an observed study, the immediate factual description is the retained sample meeting the stated estimated-score rule.
The population interpretation depends on the restriction rule. Symmetric trimming generally limits the analysis to units within the specified score interval. A treatment-specific support restriction may retain a different covariate-defined subset in each group, so the report should explain the rule and the resulting target population rather than implying that every restriction targets the same estimand. The earlier material on trimming and target populations provides the conceptual distinction; here the reporting task is to name the population implied by the actual rule.
Report how the restriction changes the composition relevant to the scientific question. For example, if high scores are predominantly associated with treated units, excluding that tail means the restricted analysis provides less information about units who were very likely to receive treatment. Do not claim that the retained sample represents all eligible patients unless that claim is justified. Nor should “better overlap” be taken to mean “more representative”: a restriction can improve empirical comparability while narrowing the population.
A short population statement can be concrete: “After restricting estimated scores to 0.10–0.90, the analysis pertains to eligible units with estimated scores in that interval, rather than all eligible units; it excludes a larger fraction of treated units from the high-score tail.” If the scientific target is a specific subgroup, name it. If the analysis does not support a clear population interpretation, state that limitation instead of implying one.
Worked Example: Interpreting the Retained Population
Suppose the symmetric restriction in the previous example is used, and the high-score treated units tend to have severe disease. The analysis retains 80 of 100 treated units and 105 of 120 controls. The restriction excludes 18 treated units above 0.90, compared with only 3 controls above that bound.
The arithmetic shows that the rule removes 18% of the original treated group from the upper tail, while it removes \(3/120=0.025\), or 2.5%, of controls from that tail. A reasonable description is: “The restricted analysis excludes estimated scores above 0.90 in both groups, disproportionately removing treated units; because high-score treated units in this hypothetical sample tend to have severe disease, the retained sample represents a less treatment-prone and less severe subset than the original treated group.”
This statement describes the sample consequence, not a causal conclusion about severity or treatment effects. It does not establish that every excluded unit had severe disease, and it does not prove that the retained sample is representative of a broader population. Those stronger claims would require additional evidence and a clearly defined target population.
Sensitivity, Diagnostics, and Alternatives
Threshold choice can materially alter both retained sample size and the population represented. Report a small, interpretable sensitivity analysis when reasonable alternative thresholds are available. For each rule, show retained and excluded counts by group and explain whether the substantive target changes. Do not select the rule because it produces a preferred outcome estimate. Outcome-guided trimming compromises the design-stage rationale for restriction and makes the reported analysis difficult to interpret independently of the observed outcome.
Useful warning signs include subgroup cells with no units from one treatment group, very small group counts, an apparent overlap conclusion driven only by extreme observations, and inconsistent counts between the stated denominators and sample flow. Check that each group’s retained and excluded counts sum to its original score-analysis count. Recheck the reported support after any score-model change, since a revised model can change both score distributions and which units meet the rule.
A restriction is not the only response to limited support. Depending on the scientific question and design, alternatives include retaining the full sample while explicitly limiting claims, changing the target population by a justified design restriction, or using an approach that handles the observed score distribution differently. These choices have different assumptions and targets; reporting should not present them as interchangeable fixes. No empirical plot or threshold can repair a genuine absence of support for the desired comparison.
Common Reporting Mistakes
- Reporting only the combined sample size. Give counts and denominators separately for treated and control units, including exclusions and retained counts.
- Equating overlapping ranges with adequate support. Ranges do not show how observations are distributed within those ranges, and they can hide sparse or unsupported subgroups.
- Leaving the threshold ambiguous. Give numerical bounds, the score used, and whether boundary values are retained.
- Calling the restricted sample representative of everyone enrolled. Describe the narrower population implied by the restriction and any important group-specific losses.
- Calling observed overlap proof of positivity. State what the sample diagnostics show, and distinguish that evidence from a population assumption.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A study has 80 treated units, of whom 8 fall below a stated lower score bound, and 100 controls, of whom 15 fall below it. How many are retained in each group, and what fraction of each group is excluded?
- Why is it insufficient to report that the treated and control score ranges overlap?
- A report states that scores from 0.05 through 0.95 were retained. What additional rule detail should be made explicit?
- If a common restriction removes many high-score treated units but few high-score controls, what should the report say about the resulting population?
Solutions. (1) Treated: \(80-8=72\) retained, with \(8/80=0.10\), or 10% excluded. Controls: \(100-15=85\) retained, with \(15/100=0.15\), or 15% excluded. (2) Overlapping ranges do not reveal density, sparse tails, or subgroup gaps, and do not establish population positivity. Report group-specific distributions and relevant subgroup support. (3) State whether the bounds are inclusive, which score is used (for example, estimated score), and how many units are excluded and retained in each treatment group. (4) State the group-specific counts and explain that the restricted analysis represents a population with fewer high-score treated units than the original sample; do not claim it still represents all eligible units without justification.