Why Replacement Matters When Controls Are Scarce
When the treated group is larger than the comparison group, matching without replacement can force some treated individuals to be paired with distant controls. Nearest-neighbor matching with replacement avoids that particular trade-off: each treated individual is matched to the closest available control, and a control can be selected more than once. The cost is dependence among matched contrasts that share a control outcome. A sound analysis must record that reuse and account for it in the estimate and its uncertainty.
Let \(A_i=1\) indicate treatment and \(A_i=0\) indicate control, let \(X_i\) denote measured pretreatment covariates, and let \(Y_i\) be the outcome. The propensity score is \(e(X_i)=P(A_i=1\mid X_i)\). For this tutorial, the target is the average treatment effect on the treated, or ATT: the average difference between treated individuals’ outcomes under treatment and the outcomes those same individuals would have had without treatment.
Assumptions and the Estimator
A causal interpretation requires assumptions beyond finding close scores. Consistency means each person’s observed outcome equals the potential outcome under the treatment actually received. Conditional exchangeability requires \(Y(0),Y(1)\) to be independent of treatment given the measured covariates \(X\). Positivity requires a nonzero probability of each treatment level for covariate patterns represented among the treated. In addition, the propensity-score model and matching procedure must adequately represent the relevant covariate structure. Matching cannot balance important unmeasured confounders by construction.
Write \(m(i)\) for the selected control and \(Y_{m(i)}\) for that control’s observed outcome. The matched ATT estimate is the treated mean minus the matched, weighted control mean. Equivalently, the control weight \(a_j\) is the fraction of treated individuals matched to control \(j\). This equivalence makes clear why a control used repeatedly contributes repeatedly to the estimated counterfactual mean.
The match should be based on pretreatment information, not on outcomes. Before interpreting the estimate, inspect the score distances and covariate balance after applying the matching weights. Report how many distinct controls were used, each control’s reuse count, the largest count, and a summary of the resulting weight concentration. A large treated sample does not compensate for an effective comparison group represented by only one or two controls.
Worked Example: Matching Four Treated Individuals to Two Controls
Worked Example: Matching Four Treated Individuals to Two Controls
Suppose a hypothetical study has four treated individuals and two controls. A propensity-score model, estimated using pretreatment covariates, gives the following scores. Matching uses absolute score distance and permits replacement.
| Group | Individual | Propensity score | Age (years) |
|---|---|---|---|
| Treated | T1 | 0.42 | 58 |
| Treated | T2 | 0.46 | 62 |
| Treated | T3 | 0.50 | 60 |
| Treated | T4 | 0.55 | 64 |
| Control | C1 | 0.40 | 57 |
| Control | C2 | 0.48 | 63 |
The nearest matches are T1 to C1 (distance \(0.02\)), T2 to C2 (distance \(0.02\)), T3 to C2 (distance \(0.02\)), and T4 to C2 (distance \(0.07\)). The matched sample therefore uses C1 once and C2 three times. With \(n_T=4\), their aggregate weights are \(a_1=1/4=0.25\) and \(a_2=3/4=0.75\).
For a simple balance check, the treated mean age is \((58+62+60+64)/4=61\) years. The weighted matched-control mean is \(0.25(57)+0.75(63)=61.5\) years, so the treated-minus-weighted-control difference is \(-0.5\) years. Before matching, the control mean was \((57+63)/2=60\) years, giving a difference of \(1\) year. This one covariate looks closer after matching, but balance must be assessed across all important pretreatment covariates, not inferred from age alone.
Estimate the ATT and Account for Reuse in Inference
The following outcome calculation uses the same hypothetical matches. Suppose the outcome is improvement in a symptom score, measured in points, with larger values indicating more improvement. The treated outcomes are 8, 7, 10, and 9 points; the control outcomes are 5 points for C1 and 6 points for C2. The matched control mean and the ATT estimate are
Here the estimate says that, under the causal assumptions and for the matched treated population, the estimated average improvement under treatment exceeds the estimated untreated improvement by 2.75 points. It is not a conclusion that matching itself eliminates confounding.
The squared aggregate weights are the key correction for reuse. In this example, \(\sum_j a_j^2=0.25^2+0.75^2=0.625\). If the two control outcomes are independent with a common variance, the variance of their weighted mean is \(0.625\sigma_C^2\). Treating the four matched contrasts as independent would fail to account for the repeated C2 outcome.
Worked Example: A Model-Based Interval That Reflects Reuse
Continue with the outcome values above. The treated sample variance is \(s_T^2=[(8-8.5)^2+(7-8.5)^2+(10-8.5)^2+(9-8.5)^2]/3=5/3=1.667\) square points. The control sample variance is \(s_C^2=[(5-5.5)^2+(6-5.5)^2]/1=0.5\) square points.
As a small-sample model-based illustration, additionally assume independent normally distributed outcomes with a common variance in treated and control groups, conditional on the fixed matches. The pooled variance estimate is \[ s_p^2=\frac{(n_T-1)s_T^2+(n_C-1)s_C^2}{n_T+n_C-2} =\frac{3(1.667)+1(0.5)}{4}=1.375. \] The estimated variance of the matched contrast is \(s_p^2(1/n_T+\sum_j a_j^2)\), so \[ \widehat{\operatorname{Var}}(\widehat{\mathrm{ATT}}) =1.375(1/4+0.625)=1.203,\qquad SE=\sqrt{1.203}=1.097\text{ points}. \]
As a rough \(t\)-based, model-based illustration—not an exact 95% \(t\) interval—using \(df=4\) and \(t_{0.975,4}=2.776\) gives \(2.75\pm2.776(1.097)=2.75\pm3.045\), or approximately \((-0.295,\,5.795)\) points. The unequal control weights mean the weighted control mean and the control contrast underlying the pooled variance estimate are not independent, so the stated assumptions do not justify exact \(t\) calibration. This heuristic interval is wide and includes zero; with only two distinct controls, the common-variance assumption and the adequacy of a small-sample model deserve particular scrutiny.
Worked Example: Reuse Can Hide a Support Problem
Worked Example: Reuse Can Hide a Support Problem
Consider a different hypothetical study with treated scores \(0.70, 0.78, 0.84\) and control scores \(0.30, 0.52\). Nearest-neighbor matching with replacement assigns all three treated individuals to the control with score \(0.52\): the distances are \(0.18, 0.26,\) and \(0.32\). That control has aggregate weight \(1\), and the other control has weight \(0\).
Suppose treated outcomes are 12, 10, and 11 points, and the selected control outcome is 7 points. The arithmetic estimate is \((12+10+11)/3-7=11-7=4\) points. But the estimate does not establish a credible treated-versus-control comparison: every treated score is substantially above the selected control score, all the counterfactual information comes from one person, and there may be little relevant overlap.
The concentration is summarized by the effective number of controls, defined as \(1/\sum_j a_j^2\). Here it is \(1/1^2=1\), not three. A small standard error calculated as if the three matched pairs contained three independent controls would be misleading. The score distances and covariate balance indicate that the numerical estimate should not be presented as a well-supported causal effect without addressing the lack of comparable controls.
Diagnostics, Alternatives, and Limitations
A useful report makes the matching structure visible rather than presenting only the final estimate. At minimum, document the matching score and distance rule, the number of treated individuals and distinct controls, reuse counts or their distribution, and weighted balance on important covariates. Inspect score distributions and the largest treated-to-control distances. Also check whether the weighted comparison is dominated by one control; \(\sum_j a_j^2\) and its reciprocal summarize this concentration, though neither replaces covariate-balance assessment.
Identify the ATT target, the propensity-score distance used, and whether controls may be reused.
List each treated match or summarize control reuse counts; verify the aggregate control weights sum to one.
Report score distances and weighted covariate diagnostics, and identify extreme reuse or weak comparability.
Calculate the weighted contrast, use an uncertainty method that reflects repeated controls, and state its assumptions and limitations.
Matching without replacement avoids repeated control outcomes but may assign less similar controls when the comparison group is small. Full matching and optimal matching use different allocation structures; weighting uses a broader set of observations but can also produce concentrated weights. Regression adjustment can supplement matching, but it does not repair a serious lack of overlap or unmeasured confounding. The choice among these approaches should be guided by the target population, balance, support, and transparent diagnostics, not by which produces the most favorable estimate.
The variance derivation above is conditional on fixed matching weights and assumptions about outcome dependence. It does not automatically incorporate propensity-score estimation, selection of the matching rule, or all sources of uncertainty in a complex design. In clustered data, or whenever treated and control outcome errors are dependent, the cross-mean covariance may be nonzero and the simple independent-groups formula is insufficient. With very few distinct controls, variance estimates are unstable; report that limitation rather than treating the nominal number of matched pairs as the number of independent comparisons.
Common Mistakes
- Counting matches as independent controls. Reusing one control three times does not create three independent control outcomes. Aggregate its weight before calculating uncertainty.
- Dropping the covariance term without a reason. The variance of a difference includes minus twice the covariance of its two means. State why that covariance is zero or use an approach that handles dependence.
- Assuming a close score proves balance. Check the measured covariates after matching; score distances alone do not verify balance on every relevant covariate.
- Interpreting a computed contrast as automatically causal. The estimate depends on exchangeability, positivity, consistency, and adequate measurement and modeling of confounders.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Five treated individuals are matched to controls with reuse counts 2, 2, and 1. What are the three aggregate control weights?
- For weights \(0.4\) and \(0.6\), calculate the effective number of controls.
- Why does a variance formula for the matched ATT need a cross-covariance term in general?
- All treated individuals are matched to one control, despite large score distances. What should be reported before interpreting the ATT?
Solutions. (1) Divide each reuse count by five: the weights are \(0.4, 0.4,\) and \(0.2\), which sum to one. (2) The effective number is \(1/(0.4^2+0.6^2)=1/0.52=1.923\) controls. (3) The variance of a difference is the first mean’s variance plus the second mean’s variance minus twice their covariance; omitting that term requires a zero-covariance assumption or another justified dependence model. (4) Report the distances, reuse count, weighted balance, and concentration of control weights; large distances and one-control dependence signal weak support, so the numerical ATT should not be presented as a well-supported causal comparison without addressing those limitations.