Why Operational Details Matter
A statement such as “we matched on the propensity score” does not uniquely describe an analysis. It leaves open whether the score itself or its logit was used, how many controls were sought per treated unit, whether controls could be reused, and what happened when no eligible match was found. Those choices can change the analysis sample and the resulting estimates. Reporting them is therefore part of defining the analysis, not merely a software detail.
The same principle applies to weighting. Naming a weighting approach is not enough if the actual weight formula, normalization, and handling of unusually large weights are omitted. A reader needs enough information to reproduce the values assigned to units and to understand any exclusions or modifications.
The propensity score is \(e(X)=P(A=1\mid X)\), where \(A=1\) denotes treatment and \(X\) is the set of measured covariates used in the propensity-score model. The model choices themselves should be reported as described in the earlier material on propensity-score model choices. Here the focus is what was done with the resulting scores. Reproducibility does not establish that the model is appropriate or that causal identification assumptions hold. A causal interpretation still relies on assumptions such as consistency, conditional exchangeability given measured covariates, and positivity for the contrast of interest.
What to Specify for Matching
For matching, report the matching method and its operating rules rather than only its general label. At minimum, state whether matching was nearest-neighbor, optimal, or another specified procedure; the matching ratio; whether matching was with or without replacement; the distance measure; and any caliper or exact-matching restrictions. If the algorithm processes units in an order, describe that order and how ties were resolved. Give the software, package and version, and any nondefault options that affect the result.
A distance measure may be the absolute difference in estimated propensity scores, the absolute difference in their logits, or a distance combining several covariates. These are not interchangeable. For example, if the distance is the absolute logit-score difference, define \(d(i,j)=|\operatorname{logit}\{\hat e(X_i)\}-\operatorname{logit}\{\hat e(X_j)\}|\). State any transformations and the scale used for the caliper. “A 0.2 caliper” is incomplete: 0.2 units of what, and measured relative to which standard deviation?
A common specification is a caliper equal to 0.2 times the standard deviation of the estimated logit propensity scores. If that standard deviation is 0.80 in the stated analysis sample, the caliper is \(0.2(0.80)=0.16\) logit units. A pair is eligible only if its distance is no greater than 0.16. To reproduce that rule, the report should also say how the standard deviation was calculated and on which sample.
Worked Example: Reporting Greedy 1:1 Matching
Suppose a hypothetical analysis uses nearest-neighbor 1:1 matching without replacement on the logit propensity score. The stated standard deviation of the logit scores is 0.80, so the caliper is \(0.2(0.80)=0.16\). Treated units are processed in the order T1, T2, T3, T4. The following are the logit scores:
Treated units: T1 = 0.10, T2 = 0.24, T3 = 0.49, T4 = 0.85. Control units: C1 = 0.08, C2 = 0.22, C3 = 0.39, C4 = 0.54. A control already matched cannot be used again.
For T1, the distances to C1 and C2 are \(|0.10-0.08|=0.02\) and \(|0.10-0.22|=0.12\). Both are within the caliper, so nearest-neighbor matching assigns C1. For T2, C1 is no longer available; C2 is at distance \(|0.24-0.22|=0.02\), so T2 is matched to C2.
For T3, C3 is at distance \(|0.49-0.39|=0.10\), and C4 is at distance \(|0.49-0.54|=0.05\). Both satisfy the caliper, so T3 is matched to the nearer available control, C4. T4 has no remaining control within 0.16: its distance to C3 is 0.46. T4 is therefore unmatched and excluded from this matched analysis under the specified rule.
A reproducible report would state the 1:1 greedy nearest-neighbor algorithm, logit-score distance, 0.16 caliper, no-replacement rule, treated-unit order, and the handling of unmatched treated units. If the order or tie rule were left unstated, a different implementation could produce different pairs.
This example also illustrates why “nearest neighbor” alone is not a complete algorithm. Greedy matching makes a locally nearest assignment at each step; the processing order can affect later assignments. An optimal matching algorithm instead seeks a minimum total distance subject to its constraints. A report should name which procedure was used rather than imply that the two are equivalent.
What to Specify for Weighting
For weighting, write the formula that assigns a weight to each treatment group, and identify the score used in it. For example, inverse-probability weights for an average treatment effect use:
Thus, a treated unit receives \(1/\hat e(X_i)\), while a control unit receives \(1/\{1-\hat e(X_i)\}\). Other weighting targets use different formulas. For example, weights for an average treatment effect among treated units are \(w_i=A_i+(1-A_i)\hat e(X_i)/\{1-\hat e(X_i)\}\). Reporting only “inverse-probability weighting” does not tell the reader which formula was applied.
Also state whether weights were normalized, and how. Normalization might divide each weight by the mean weight in a treatment group, or rescale weights so they sum to a specified group size. Those procedures change the numerical weights and should not be left implicit. If weights were truncated, capped, or units were trimmed, report the exact rule, its threshold, whether it was applied within groups or overall, and how many weights or units it affected. If no such action was taken, say so.
Worked Example: Calculating and Reporting ATE Weights
Consider a hypothetical dataset with two treated units and two control units. Their estimated propensity scores are 0.20 and 0.60 for the treated units, and 0.25 and 0.80 for the control units. Using the ATE formula above, the treated weights are:
For the treated unit with \(\hat e(X_i)=0.20\), \(w_i=1/0.20=5.000\). For the treated unit with \(\hat e(X_i)=0.60\), \(w_i=1/0.60=1.667\).
The control weights use \(1/\{1-\hat e(X_i)\}\). For the control with score 0.25, \(w_i=1/(1-0.25)=1/0.75=1.333\). For the control with score 0.80, \(w_i=1/(1-0.80)=1/0.20=5.000\). The four raw weights are therefore 5.000, 1.667, 1.333, and 5.000, in the order listed.
A transparent methods description would identify this as raw ATE inverse-probability weighting, give the formula, and state whether weights were normalized or modified. In this example, suppose no normalization, truncation, or trimming was applied. The report should say so; it should not suggest that the large weights were removed.
Weight size is a procedural issue to document, but a threshold does not have a universal default that makes it harmless. A decision to cap or trim weights changes the analysis and must be justified and reported, not silently introduced after inspecting results. The report should distinguish a rule specified in advance from a later sensitivity analysis.
Handling Exceptions and Making the Record Auditable
The ordinary case is not the entire procedure. State what happened when a treated unit had no eligible control, when a requested matching ratio could not be achieved, or when a score produced a weight beyond a prespecified limit. Explain whether units were discarded, retained with fewer matches, or handled under another explicit rule. Report counts that let readers reconcile the procedure: numbers entering matching, numbers matched and unmatched, and the number of unique controls used when replacement is allowed. These are procedural counts, not substitutes for later balance or overlap diagnostics.
Worked Example: Matching With Replacement and Incomplete Sets
Suppose a hypothetical study specifies 1:2 nearest-neighbor matching on the absolute propensity-score difference, with replacement and a maximum distance of 0.10. Two treated units have scores 0.40 and 0.70. Three controls have scores 0.42, 0.65, and 0.95. For each treated unit, choose the two nearest controls that are within the distance limit; controls may be reused.
For the treated unit with score 0.40, the distances are \(|0.40-0.42|=0.02\), \(|0.40-0.65|=0.25\), and \(|0.40-0.95|=0.55\). Only the first control is within 0.10, so this treated unit has one match rather than two.
For the treated unit with score 0.70, the distances are \(|0.70-0.42|=0.28\), \(|0.70-0.65|=0.05\), and \(|0.70-0.95|=0.25\). Again only one control is eligible, the control with score 0.65. Because replacement is allowed, this result is not blocked by any earlier assignment; in this particular example, the two matched controls are different.
The report should not call this a complete 1:2 match: one eligible control was found for each treated unit, so the realized sets are 1:1. It should state that the target ratio was 1:2, the caliper was 0.10 score units, matching was with replacement, and units with fewer eligible controls were retained with fewer matches (if that is the adopted rule). It should also report the number of unique controls, since the number of matched control records can exceed the number of distinct control units when reuse occurs.
For reproducibility, identify software and version, relevant package or procedure, and options that affect matching or weights. If the method uses random ordering, state the seed and the ordering rule, while recognizing that a seed alone may not guarantee identical results across software versions. Provide code or a sufficiently detailed algorithm when the procedure is custom. A clear report separates prespecified decisions from implementation details and from deviations made during analysis.
Common Pitfalls and Comparison of Approaches
- “We used propensity-score matching.” This omits the algorithm, distance, ratio, replacement rule, caliper, and exception handling. Name each choice.
- “Extreme weights were removed.” This does not identify what counted as extreme or how removal occurred. Give the numeric rule, the units affected, and the disposition of their observations.
- “The software defaults were used.” Defaults can depend on software version and may not be evident to readers. Report the procedure and consequential settings, then provide version or code details.
- “A 1:2 match was conducted.” That can describe the requested ratio, not necessarily the achieved ratio. State how incomplete matches were handled and report the realized counts.
Matching constructs a set of paired or grouped observations under explicit distance and eligibility rules; weighting assigns each unit a numerical contribution under a formula. Matching can leave units unmatched, while weighting can give some units large contributions. Neither approach is automatically preferable, and neither is reproducible from its name alone. The choices should be reported in terms of the method actually implemented. Balance assessment and overlap assessment answer separate questions and should be reported separately rather than implied by a detailed algorithm description.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A report says “nearest-neighbor matching with a 0.2 caliper.” Name three details needed to make this statement reproducible.
- For an ATE weighting analysis, a treated unit has \(\hat e(X_i)=0.40\). What is its raw inverse-probability weight?
- A control unit is matched to three treated units in a with-replacement procedure. What should the report clarify about reuse and the number of controls?
- An analyst caps all weights at 10 but reports only the capped analysis. What information about this decision is missing?
Solutions. (1) Specify the distance scale and caliper units, the matching ratio, and whether matching is with or without replacement; also report algorithm/order and how unmatched units are handled. For example, 0.2 logit units differs from 0.2 standard deviations of the logit score. (2) Since the unit is treated, \(w_i=1/\hat e(X_i)=1/0.40=2.500\). (3) State explicitly that controls could be reused, how matches were assigned, and both the number of matched control records and the number of unique control units. (4) State the exact cap, the formula and stage at which it was applied, how many weights were affected, and the rule for any resulting unit handling; also distinguish whether the rule was prespecified or examined as a sensitivity analysis.