From Estimated Scores to a Defensible Comparison
A trimming analysis is not just a choice of a cutoff. It is a sequence of design decisions: estimate treatment probabilities from pretreatment covariates, inspect where treated and untreated units have comparable scores, check whether the apparent overlap hides poorly supported subgroups, and state precisely who is removed. The analysis should be conducted without using observed outcomes to choose the score model or threshold. Otherwise, the design can become tailored to the outcomes it is meant to analyze.
Let \(A=1\) denote treatment and \(A=0\) denote control, let \(X\) be the measured pretreatment covariates, and define the propensity score as \(e(X)=P(A=1\mid X)\). In practice, the score is estimated from a model, producing \(\hat e(X)\). A score near zero means the model assigns a low treatment probability to a unit with those covariates; a score near one means it assigns a high probability. Neither score alone establishes that the unit's observed treatment was random or that a causal effect is identified.
Causal interpretation still relies on the usual requirements for an observational treatment comparison, including consistency, conditional exchangeability given the measured covariates, and positivity for the population being targeted. Trimming can remove regions with weak empirical support, but it cannot establish that all confounders were measured or correct a misspecified score model. It also does not make an unadjusted comparison causal simply by removing extreme scores.
Estimate and Inspect the Score Model
A common starting point for binary treatment is logistic regression. For covariates \(X_1,\ldots,X_k\), the specified model has the form
The coefficients are estimated using treatment and pretreatment covariate data, not outcomes. The analyst should examine whether important nonlinearities or interactions are plausible, whether the fitting procedure converged, and whether separation or sparse covariate patterns are causing unstable estimates. A good-looking score histogram is not proof that the model correctly represents treatment assignment. Conversely, changing the model can change the score distributions and the units that a fixed threshold retains, so overlap must be reassessed after a material model change.
Worked Example: Calculating Scores From a Fitted Model
Suppose a hypothetical study models receipt of a program using age in decades, an indicator for high baseline severity, and an indicator for rural residence. A fitted logistic model is \(\hat\eta=-2.2+0.8(\text{age in decades})+1.1(\text{high severity})+0.7(\text{rural})\). The following calculations illustrate how the fitted model assigns scores; they do not describe real study participants.
For a 40-year-old without high severity who lives in an urban area, the covariate values are 4, 0, and 0. Thus \(\hat\eta=-2.2+0.8(4)=1.0\), and \(\hat e=\exp(1)/(1+\exp(1))=0.7311\). The model assigns this profile an estimated treatment probability of about 73.1%.
For a 60-year-old with high severity in an urban area, \(\hat\eta=-2.2+0.8(6)+1.1=3.7\), so \(\hat e=\exp(3.7)/(1+\exp(3.7))=0.9759\). This is a near-one score: the fitted model predicts treatment is very likely for this profile.
For a 30-year-old without high severity who lives in a rural area, \(\hat\eta=-2.2+0.8(3)+0.7=0.9\), giving \(\hat e=\exp(0.9)/(1+\exp(0.9))=0.7109\). These values are scores implied by the specified fitted model, not observed treatment probabilities known without error. Their usefulness depends in part on the model's adequacy and the covariates included.
Support Plots and Subgroup Checks
Plot estimated scores separately for treated and control units. Overlaid histograms, density displays, empirical distribution plots, and rug plots can reveal different features: histograms show counts in chosen bins, while rugs show individual score locations. Always report group sample sizes, since separately normalized density curves can look similar even when one group has far fewer observations. A plot is a diagnostic, not a formal proof of positivity or balance.
A full-sample display can conceal gaps within important groups. For example, the treated and control score distributions may overlap overall because low-risk controls overlap with one treated subgroup and high-risk treated units overlap with a different control subgroup. That pattern does not guarantee useful comparisons within either risk group. Inspect support within substantively important covariate subgroups, and consider whether the subgroup has enough units from both arms to support the intended comparison. The choice of subgroups should be driven by the design and scientific question, not by which partition gives a preferred outcome estimate.
Worked Example: Reading Score Distributions and Checking a Subgroup
Consider a hypothetical sample with 100 treated and 100 control units. The table gives the number of estimated scores in each interval. Counts in the intervals are complete and nonoverlapping; no observation lies exactly on a boundary.
| Estimated-score interval | Treated | Control |
|---|---|---|
| Below 0.10 | 2 | 15 |
| 0.10 to below 0.20 | 8 | 20 |
| 0.20 to below 0.40 | 20 | 25 |
| 0.40 to below 0.60 | 30 | 20 |
| 0.60 to below 0.80 | 25 | 12 |
| 0.80 to below 0.90 | 10 | 6 |
| 0.90 or above | 5 | 2 |
| Total | 100 | 100 |
The table suggests that both arms occur throughout the middle intervals, while the control group has many more observations below 0.10 and the treated group has more at or above 0.90. It is a grouped summary, so it does not identify exact support boundaries or show whether scores are clustered within an interval. Plotting individual scores and examining finer intervals would help assess those details.
Now divide the same hypothetical sample by baseline severity. Among treated units, the low-severity subgroup has 60 units and 2 scores outside 0.10–0.90; the high-severity subgroup has 40 units and 5 scores outside. Among controls, the corresponding counts are 60 with 4 outside and 40 with 13 outside. These figures sum to the full-sample exclusions of 7 treated and 17 controls. The high-severity subgroup therefore has less retained support, especially among controls: only \(27/40=67.5\%\) of its controls remain, compared with \(56/60=93.3\%\) of low-severity controls. An overall plot alone would not communicate this difference.
Why the Score Can Support Comparison—and Where It Cannot
The balancing property explains why the true propensity score is useful for organizing support diagnostics. At an interior score value \(q\), the treatment probability is the same for every covariate profile with that score. Consequently, treated and control units within that score stratum have the same covariate distribution, under the true score. This statement does not say that the estimated score automatically balances covariates in a finite sample.
The endpoint qualification matters in practice. A score of exactly zero or one is not evidence of excellent overlap: it indicates that one arm has no probability under the true assignment mechanism at that score. Estimated scores close to an endpoint are also a warning that comparisons may rely on a small number of atypical units or on model extrapolation. Trimming rules should be justified in terms of the comparison that remains possible, not by claiming that the score property supplies information from an absent arm.
Choose a Rule, Then Account for Its Consequences
A threshold should be stated as a design rule, along with its rationale. For example, an analyst might retain units with \(0.10\leq\hat e(X)\leq0.90\), or use treatment-specific restrictions based on empirical support. These are not interchangeable choices. A symmetric score window applies the same numerical limits to both groups; treatment-specific restrictions can retain different score regions by arm. Neither is universally preferable. The decision depends on the support pattern, the target population, and whether the retained sample can support the planned comparison.
Do not select the cutoff by trying several values and retaining the one that produces the most favorable outcome estimate or statistical significance. Instead, compare plausible rules using treatment-specific score displays, subgroup support, and retention. If the threshold is chosen after examining these design diagnostics, say so and show sensitivity to reasonable alternatives when useful. A threshold is a practical restriction, not a universal boundary at which causal inference suddenly becomes valid.
Worked Example: Calculating Retention and Describing the Restricted Population
Apply the prespecified hypothetical rule \(0.10\leq\hat e(X)<0.90\) to the 200-unit sample above. The grouped table shows that 2 treated units below 0.10 and 5 at or above 0.90 are excluded. Thus \(100-2-5=93\) treated units remain, for a treated retention rate of \(93/100=93.0\%\). For controls, \(15+2=17\) are excluded, leaving \(100-17=83\), or \(83.0\%\).
In total, \(93+83=176\) units remain. The overall retention rate is \(176/200=88.0\%\), and \(24/200=12.0\%\) are excluded. Reporting only the overall percentage would hide that control retention is lower, so the arm-specific rates are essential. The subgroup calculations also show that high-severity control retention is only 67.5%.
The retained sample targets a population restricted by the estimated-score rule. It is not automatically the original study population, and a causal effect for the retained population still requires the relevant identification assumptions and an analysis appropriate to its target estimand. In particular, trimming both arms does not by itself specify whether the desired effect is an average for all retained units, retained treated units, or another population. State that target explicitly. Record the rule, score model, counts excluded from each arm, subgroup implications, and any changes to the analysis population.
Review the Retained Design and Recognize Failure Modes
After restriction, review whether the retained data still support the intended comparison. Check the score distributions again and inspect covariate distributions in the retained sample. If important covariate differences remain, the restriction has not resolved them; use an appropriate adjustment strategy rather than treating overlap as proof of balance. Formal balance measures and displays provide a separate design diagnostic.
Several warning signs deserve attention. A fitted model may assign extreme scores because of sparse categories, separation, or an overly rigid functional form. A score plot may show apparent marginal overlap while a key subgroup has almost no units in one arm. A chosen cutoff may discard a large share of a scientifically important group and thereby change the question. Finally, an analyst may report a single overall retention rate while omitting which arm or subgroup bears most exclusions. These are reasons to revisit the model, the target population, or the design—not reasons to hide the restriction.
Overlap analysis is different from relying on extrapolation into regions with little or no support. It is also different from changing the score model to improve fit to outcomes, which risks outcome-guided design. A defensible report explains the score specification, the plots and subgroup checks used, the threshold rule, the retention arithmetic, and the population to which the later comparison applies. Keep outcome results out of the threshold-selection rationale.
Check Your Understanding
Answer each question using the definitions and calculations above. Solutions follow the questions.
- Why should observed outcomes not determine the score threshold in a design-stage trimming analysis?
- In the hypothetical score table, how many controls remain under the rule \(0.10\leq\hat e(X)<0.90\), and what percentage of controls are excluded?
- Why does an overall overlap plot not settle whether the high-severity subgroup has adequate support?
- What is the interpretation of trimming 12.0% of the sample in the worked example?
Solutions. (1) Choosing a threshold to favor an outcome estimate uses the outcome to shape the design and can produce outcome-guided selection; choose and justify the rule using treatment and pretreatment covariate information. (2) There are \(100-15-2=83\) retained controls. The excluded proportion is \(17/100=17.0\%\). (3) The overall plot combines subgroups, so support in one subgroup can mask a lack of observations from one treatment arm in another; inspect subgroup-specific counts and score distributions. (4) It means 24 of the 200 hypothetical units were excluded by the stated estimated-score rule. It does not mean that the retained sample represents the original population, nor that the causal assumptions are thereby satisfied.