Why the Timing of a Trimming Decision Matters
Trimming can limit comparisons to people whose observed covariates indicate a reasonable chance of receiving either treatment. But the threshold is not merely a technical setting: changing it can change both which people contribute to the analysis and the population to which the result applies. A choice made after examining treatment outcomes can therefore affect the conclusion in two ways at once.
The central safeguard is to make trimming decisions using the study design and covariate information, rather than selecting the rule that produces the most appealing treatment-effect estimate, confidence interval, or p-value. This does not make a causal analysis automatically valid. It reduces one avoidable source of outcome-dependent selection while leaving other assumptions and diagnostics essential.
For example, a symmetric rule might retain people with \(c\le e(X)\le 1-c\), for a prespecified threshold \(c\). More generally, a rule can define a retained set \(R=1\) using scores or covariates. The treatment-effect target after trimming concerns the retained population, not automatically the original study population. In potential-outcome notation, a target might be the average treatment effect among retained people, \(E[Y(1)-Y(0)\mid R=1]\). If the retained set changes, the target can change too.
Causal interpretation still depends on the relevant assumptions, including consistency, conditional exchangeability given measured pretreatment covariates, and adequate positivity in the population being analyzed. The score model and effect estimator must also be appropriate for their roles. Trimming may address unsupported regions by restricting the target; it does not establish these assumptions or remove bias from unmeasured confounding.
How Outcome-Guided Choices Distort Conclusions
An analyst might try several thresholds, calculate the estimated effect for each, and retain the one that yields statistical significance, a larger effect, a preferred direction, or a narrower interval. This is not equivalent to carrying out one prespecified analysis. The chosen result has been selected partly because of the outcome data, so ordinary uncertainty calculations for a single fixed analysis do not account for the selection process.
There is a second problem even if the estimates were calculated without random error: different thresholds can retain different people. An estimate for people with scores between 0.05 and 0.95 and an estimate for people between 0.20 and 0.80 may describe different target populations. A difference between those estimates can reflect a change in the target, not simply a more accurate estimate of one fixed effect. Choosing the threshold after seeing which target gives the preferred result makes the scientific question itself outcome-dependent.
Worked Example: Three Thresholds, Three Different Conclusions
Consider a hypothetical study with 1,000 people and a continuous outcome measured in points. An analyst uses the same covariate-adjusted estimation procedure at three symmetric thresholds. The table gives illustrative adjusted outcome means for treated and control people in each retained population. Assume, for this example, that the reported standard errors are calculated for each fixed rule.
| Retained score range | Retained people | Adjusted treated mean | Adjusted control mean | Estimated difference (SE) |
|---|---|---|---|---|
| 0.05 to 0.95 | 840 | 12.4 | 10.1 | 2.3 (1.1) |
| 0.10 to 0.90 | 690 | 11.8 | 10.6 | 1.2 (0.9) |
| 0.20 to 0.80 | 430 | 11.0 | 11.3 | -0.3 (0.8) |
For the first rule, the estimated difference is \(12.4-10.1=2.3\) points. Its test statistic, using the displayed standard error, is \(2.3/1.1=2.09\). For the second rule, the difference is \(11.8-10.6=1.2\) points, and the corresponding statistic is \(1.2/0.9=1.33\). For the third, the difference is \(11.0-11.3=-0.3\) points, with statistic \(-0.3/0.8=-0.375\).
If the analyst selects the first rule because its positive result is most compelling, the selection has used outcome information. Moreover, the first and third analyses concern retained populations of 840 and 430 people, respectively; their estimates need not be estimates of the same population effect. The table is not evidence that one threshold reveals the “true” effect. It illustrates why the threshold and its target should be justified independently of which result looks best. The differences are illustrative, not causal conclusions: their causal interpretation would still require the stated identification and estimation assumptions.
A Short Justification: Selection Can Favor Noise
The selective-inference problem can be seen in a simplified setting where several candidate analyses all estimate the same null effect \(\theta=0\), and their estimates differ only because of random error. This simplification does not capture the changing target population in trimming, but it isolates the effect of choosing a result based on its value.
For two candidate estimates, when \(\theta=0\),
If the errors have mean zero, the first term has expectation zero. The second term is nonnegative and is positive on average whenever the errors are not identical. Thus, selecting the larger estimate tends to produce an estimate above the null even though neither candidate is systematically above it before selection. In real trimming analyses, the candidate estimates are usually dependent and may target different populations, so this simple derivation is not a formula for their bias. It does show why standard errors and p-values for one fixed rule do not automatically account for choosing that rule from the results.
Worked Example: Trying Multiple Tests Can Raise the Chance of a False Positive
Suppose, purely for illustration, that an analyst tests the same true null hypothesis using five independent candidate analyses, each at \(\alpha=0.05\), and reports the one with a significant result if any is significant. Under these assumptions, the chance that a single test is not significant is \(1-0.05=0.95\). Independence gives the chance that none of the five is significant as \(0.95^5\).
The chance is about 22.6%, not 5%, under this simplified setup. Candidate trimming analyses are generally correlated, so this number should not be presented as the actual error rate for a particular analysis. The lesson is narrower: choosing among several results and then using a single-analysis p-value as though the choice had not occurred does not preserve the stated 5% error rate in general.
Prespecify a Design-Based Rule
A defensible rule states how the retained population will be defined before the analyst examines the treatment outcomes. The rule may be informed by score distributions, covariate support, subject-matter knowledge, and the intended target population. A numerical cutoff is not automatically justified merely because it was written down early; its rationale and consequences should also be assessed.
Decide which population the analysis is intended to describe and what support or information problems the restriction is meant to address. State the score rule or other restriction in operational terms.
Use pretreatment variables, treatment status, and propensity-score diagnostics to evaluate whether the planned rule is workable. Do not compare outcome estimates to decide which rule to keep.
Record who is retained and excluded, and recheck whether the groups are sufficiently comparable on measured covariates for the planned analysis. If the design is inadequate, follow a prespecified contingency or revise the design using outcome-blind criteria.
Use the selected rule and estimator to analyze outcomes. If other thresholds are shown, treat them as planned sensitivity analyses and explain that they may target different retained populations.
Worked Example: Applying a Prespecified Rule Before Outcome Analysis
Suppose a hypothetical study has 500 treated and 500 control participants. Before examining outcomes, the team specifies a symmetric rule: retain participants with estimated propensity scores from 0.10 through 0.90. The design diagnostics show that 440 treated participants and 420 control participants meet the rule, so \(440+420=860\) of the original 1,000 participants are retained.
The team also specifies that it will assess covariate balance after restriction and will not choose a new threshold because of the treatment-effect estimate. If the retained design fails its prespecified adequacy criteria, the team will report that limitation and consider a documented, outcome-blind design revision before estimating the effect. This avoids using the outcome to decide who belongs in the analysis.
After the rule is fixed, suppose the estimated adjusted treated and control means are 14.2 and 12.7 points, respectively, with \(SE=1.0\) point for the estimated difference. The estimate is
Using a normal-approximation 95% interval for this hypothetical fixed-rule analysis gives \(1.5\pm1.96(1.0)\), or \((-0.46,\ 3.46)\) points. Under the model and variance calculation used for this interval, it is compatible with effects from a 0.46-point decrease to a 3.46-point increase in the retained population. It includes zero, so these data do not establish a nonzero effect at the corresponding two-sided 5% level. This interval does not account for uncertainty omitted by the estimation procedure, such as any propensity-score estimation uncertainty it fails to incorporate. Nor does prespecification alone make the estimate causal: the causal interpretation still depends on the assumptions for the retained population.
Diagnostics, Alternatives, and Limits
Warning signs of outcome-guided trimming include repeated threshold changes after viewing estimated effects; reporting only the threshold with the preferred sign, p-value, or interval; and describing alternative thresholds as “robustness checks” while omitting those with unappealing results. A selection decision can also be outcome-guided even if it is described as choosing the “best balance” when the analyst has inspected outcome estimates and those estimates influence the choice.
The main alternative is a design-first analysis: define a rule using support and covariate information, assess the resulting design, and then analyze the outcome for the population that rule retains. A planned sensitivity analysis can show results for several justified rules, provided the rules and their interpretation are clear. Such a display is not a license to choose one estimate as the answer after seeing the results. Since the retained populations may differ, describe each estimate as applying to its own target rather than treating the set as repeated measurements of one fixed effect.
If a prespecified rule produces unacceptable support or covariate balance, the response is not necessarily to proceed regardless. A design revision may be appropriate, but it should be motivated by design diagnostics, made without selecting for a desired outcome result, and followed by a clear account of how the analysis changed. Other strategies for limited overlap, including approaches that change weighting or modeling, also have assumptions and may change the target or rely more heavily on modeling. They do not remove the need to define the scientific question and examine the design.
Prespecification has limits. A rule can be arbitrary, overly restrictive, or poorly matched to the intended population. Propensity scores are estimated, and the observed score distribution is subject to sampling variability; a prespecified numerical cutoff does not make the estimated support known without error. In addition, outcome data may be necessary for the final estimation and uncertainty assessment. The key distinction is not whether outcomes are ever analyzed, but whether they are used to select the trimming decision that is then presented as if it had been fixed in advance.
Common Mistake
Wrong reasoning: “I tried several trimming thresholds, and one produced a significant, clinically favorable estimate. Since that threshold also retained some participants from both groups, I can report its ordinary p-value as confirmatory.”
Correction: Retaining both groups does not show that the comparison has adequate support, and the preferred result was selected using outcome information. The chosen rule may also target a different population from the alternatives. Report the selection honestly, avoid treating the single-analysis p-value as if the rule were fixed, and base primary interpretation on a design justified independently of the observed effect. If thresholds were prespecified as sensitivity analyses, report them as analyses of their corresponding retained populations rather than selecting a winner.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An analyst tries four thresholds and reports only the one with the smallest p-value. What is the problem with interpreting that p-value as an ordinary single-analysis result?
- Why can estimates from two trimming thresholds differ even if both analyses are conducted correctly?
- A team fixes a score threshold before seeing outcomes. Does that by itself establish a causal interpretation? Name two additional requirements.
- For five independent tests at \(\alpha=0.05\), calculate the chance of at least one false positive under the global null.
- After applying a prespecified rule, covariate diagnostics indicate that the retained groups remain poorly comparable. What is a defensible next step, and what should not drive it?
Solutions. (1) The threshold was selected using the results, so the reported p-value does not account for that selection and need not have the ordinary single-analysis interpretation. (2) The thresholds can retain different people and therefore target different populations; estimates can also vary through sampling error. (3) No. Causal interpretation still requires assumptions such as consistency and conditional exchangeability given measured pretreatment covariates, as well as adequate positivity in the retained population. (4) \(1-(1-0.05)^5=1-0.95^5=0.2262\), or about 22.6%, under the stated independence assumption. (5) Follow a documented contingency or revise the design using outcome-blind support and covariate diagnostics, then explain the resulting target and limitations. Do not select a new threshold because it yields a preferred treatment-effect estimate.