Why Threshold Choice Is a Tradeoff
Restricting an analysis to observations with propensity scores in a chosen range can remove regions where treated and control units have little common support. A tighter range may improve the plausibility of comparisons within the retained sample, but it also excludes people, can reduce representation of important subgroups, and may leave less information for estimation. The practical decision is therefore not simply “trim as much as possible” or “retain as many people as possible.” It is to compare candidate rules using design information and make the consequences of each rule visible.
The introductory material on trimming and retained populations establishes that changing a trimming rule can change the population to which an analysis pertains. Here the focus is how to evaluate that change across several criteria. A threshold should be selected using propensity-score distributions, treatment counts, subgroup coverage, and suitable precision planning—not by trying thresholds until one produces a preferred treatment-effect estimate.
The estimated score is a design diagnostic, not a guarantee that the relevant assumptions hold. A causal interpretation for the retained population still requires the relevant identification conditions, including adequate treatment overlap there and conditional exchangeability given the covariates used in the analysis. Trimming does not itself establish exchangeability, repair omitted confounding, or ensure that the score model is well specified. It also does not make a comparison informative in covariate regions that remain unsupported.
A Structured Comparison of Candidate Rules
Start with a small set of defensible candidate thresholds rather than searching an unrestricted grid for a favorable result. For each candidate, report the retained count and retention proportion overall and separately by treatment arm. Then inspect the retained score distributions or score-band counts by arm. Finally, repeat the retention summaries for subgroups that matter substantively, and consider whether the remaining counts support the intended level of precision.
Use ranges motivated by the observed score distributions and the intended design. State each rule explicitly, including whether boundary scores are retained.
For each rule, give retained counts and percentages overall and by treatment arm. A high overall retention percentage can conceal substantial loss from one arm.
Check whether both arms remain represented across the retained score range and within important subgroups. Counts by score band are more informative than a range alone.
Use retained arm sizes, and when appropriate external or blinded variance information, to gauge whether the proposed design is likely to be sufficiently informative. Do not use treatment-effect estimates to choose the rule.
No single summary captures overlap. A rule may retain observations from both arms across its stated score interval while still retaining few controls in one part of that interval. Conversely, a narrow interval may produce score distributions that look closer while removing many people. Score-band counts and plots help show this pattern; they do not replace checks of covariate support or later assessment of covariate balance.
Worked Example: Comparing Retention and Score-Band Separation
Suppose a hypothetical cohort has the following score-band counts. Consider candidate A, which retains scores from 0.05 through 0.95, and candidate B, which retains scores from 0.10 through 0.90. Assume the endpoints match the bands shown in the table.
| Estimated score band | Treated | Control | Total |
|---|---|---|---|
| Below 0.05 | 2 | 38 | 40 |
| 0.05 to below 0.10 | 8 | 32 | 40 |
| 0.10 through 0.90 | 120 | 130 | 250 |
| Above 0.90 through 0.95 | 32 | 8 | 40 |
| Above 0.95 | 38 | 2 | 40 |
There are 200 treated and 210 control observations, for 410 in total. Candidate A retains the middle three bands: \(8+120+32=160\) treated and \(32+130+8=170\) controls. Thus it retains \(330/410=0.805\), or 80.5%, overall; the arm-specific retention rates are \(160/200=80.0\%\) and \(170/210=81.0\%\).
Candidate B retains only the central band: 120 treated and 130 controls. It retains \(250/410=0.610\), or 61.0%, overall, with arm-specific retention rates \(120/200=60.0\%\) and \(130/210=61.9\%\). It therefore sacrifices 80 observations compared with A. Its central band has a treated share of \(120/(120+130)=0.480\). In A’s lower edge band, the treated share is \(8/40=0.200\); in its upper edge band, it is \(32/40=0.800\). These edge-band counts show why A’s broader interval retains regions with stronger treatment-arm separation, even though both arms appear in every retained band.
The summaries do not establish that B is preferable. They show the tradeoff: B excludes more observations from the most separated bands, while A retains substantially more of the cohort. The analyst should consider whether the remaining score and covariate support under A is adequate for the intended comparison, and what loss of population coverage under B would mean. Neither conclusion can be made from retention percentages alone.
Subgroup Coverage Is Part of the Decision
A rule can appear acceptable in the full sample while removing a large fraction of a group important to the scientific question. For that reason, calculate retention by treatment arm within pre-specified subgroups. The subgroup definition should be based on substantive considerations, not selected after inspecting which groups make a candidate threshold look favorable. These summaries reveal who is being excluded; they do not ensure adequate support for every covariate pattern within a subgroup.
Worked Example: Checking Coverage Within a Subgroup
Use the hypothetical cohort above and suppose rural residence is an important subgroup. The following counts are consistent with the overall totals.
| Group | Original treated | Original control | Retained by A: treated / control | Retained by B: treated / control |
|---|---|---|---|---|
| Rural | 60 | 70 | 40 / 55 | 20 / 40 |
| Urban | 140 | 140 | 120 / 115 | 100 / 90 |
| Total | 200 | 210 | 160 / 170 | 120 / 130 |
Under A, rural retention is \(40/60=66.7\%\) among treated people and \(55/70=78.6\%\) among controls. Under B, it is \(20/60=33.3\%\) and \(40/70=57.1\%\), respectively. Candidate B leaves both treatment arms represented among rural people, but it retains far fewer of them, particularly treated rural people. If the target question requires meaningful representation of rural participants, this loss is an important cost of B. The table does not by itself prove that A has sufficient rural overlap: rural score distributions and relevant covariate support still need inspection.
Precision as a Design Consideration
Smaller retained arm sizes commonly imply less precision, all else equal. A simple planning approximation illustrates the connection. If the outcome variances in the two arms are \(\sigma_1^2\) and \(\sigma_0^2\), and the target is a difference in arm means under independent observations, its sampling variance is approximately \(\sigma_1^2/n_1+\sigma_0^2/n_0\), where \(n_1\) and \(n_0\) are the retained arm sizes. This is a planning calculation for a simple difference in means, not a general variance formula for every adjusted causal estimator.
The implication follows directly: when the variances are held fixed, decreasing either arm size increases its contribution to the variance. But trimming may also improve the comparability of the retained groups, and the simple calculation does not quantify bias from poor overlap or the variance of a particular adjusted estimator. Precision should therefore be considered alongside support, rather than used as a substitute for it.
Worked Example: A Simple Precision Comparison
For illustration, suppose external planning information suggests an outcome standard deviation of 10 units in each arm. Use the retained counts from the first example and the simple difference-in-means approximation. The assumption of equal standard deviations is made only for this calculation; it is not inferred from the score table.
For candidate A, \(n_1=160\) and \(n_0=170\). The approximate standard error is
For candidate B, \(n_1=120\) and \(n_0=130\), so
Under these simplifying assumptions, B’s approximate standard error is \(1.266/1.101\approx 1.150\), about 15.0% larger. This quantifies a potential precision cost of the smaller retained sample; it does not show that A should be chosen. A can have greater precision under this calculation while still having poorer support near the retained boundaries. The calculation uses no observed treatment-effect estimate to select a threshold.
Justification for Comparing Counts by Arm
The basic precision calculation also explains why total retained sample size alone is inadequate. If outcome variances are equal to \(\sigma^2\), then the approximate variance is \(\sigma^2(1/n_1+1/n_0)\). For a fixed total \(n_1+n_0\), this quantity is minimized when the two arm sizes are equal. To see why, the product \((1/n_1+1/n_0)\) equals \((n_1+n_0)/(n_1n_0)\). At fixed sum, the product \(n_1n_0\) is largest when the two counts are equal, since \((n_1-n_0)^2 \geq 0\) implies \(n_1n_0 \leq (n_1+n_0)^2/4\). Thus arm imbalance can limit precision even when the overall retained count is sizable. This is a precision argument under the stated simple model, not a criterion that overrides support or target-population considerations.
Common Pitfalls and Limits
- Choosing the rule with the highest retained percentage. This ignores whether the two arms remain comparable across the retained range. Report score distributions and arm-specific band counts alongside retention.
- Choosing the rule with the most similar score distributions by itself. Similarity in a score summary does not prove covariate balance or support in every relevant covariate pattern. Examine subgroup and covariate support as well.
- Using a preferred treatment-effect estimate to pick the threshold. This makes the design decision outcome-guided and can make subsequent estimates and uncertainty assessments difficult to interpret. Set and document the rule using design diagnostics, without searching for a favorable effect.
- Assuming a narrow interval automatically solves positivity. A restriction may remove score extremes without resolving sparse treatment cells or unsupported covariate combinations that remain.
- Treating the simple standard error as the precision of any causal estimator. The formula above assumes a simple difference in means and independent observations. The actual variance depends on the estimator, design, and outcome variation.
Threshold comparisons are therefore evidence for a transparent design choice, not an automatic optimization problem with a universally best answer. The chosen rule should be justified in terms of the intended target population, observed overlap, subgroup coverage, and realistic precision needs. Report alternatives and their consequences when reasonable criteria point in different directions. Retention alone cannot tell whether the causal assumptions hold, and no threshold can recover information absent from the data.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A candidate rule retains 90% of the cohort overall, but retains 95% of treated observations and 70% of controls. What should you report and investigate before judging the rule?
- Why can a score interval that contains both treated and control observations still show concerning separation?
- Under the equal-variance planning approximation, which has the smaller variance: arm counts of 100 and 100, or 150 and 50, if the total sample size is 200?
- Give one reason a threshold should not be selected by choosing the rule that produces the most favorable treatment-effect estimate.
Solutions. (1) Report the total and arm-specific counts and retention percentages, then inspect score distributions and relevant subgroup coverage; the high overall percentage conceals unequal loss by arm. (2) Both arms may occur somewhere in the interval but be concentrated in different score bands, so the range alone does not show how much support they share. (3) With equal variance \(\sigma^2\), the variance factor is \(1/n_1+1/n_0\). It is \(1/100+1/100=0.0200\) for counts 100 and 100, and \(1/150+1/50=0.0267\) for counts 150 and 50; the balanced counts have the smaller variance. (4) Selecting on the estimated effect uses outcome information to define the design and can produce a result-dependent analysis. Thresholds should instead be justified using pre-specified design diagnostics.