Why the Weight Given to a Matched Set Matters
Matching can produce several sets, and those sets need not contain the same number of units. Once an effect has been estimated within each set, an aggregate still has to be chosen: should a small set count as much as a large one, or should larger sets contribute more? These are not merely alternative arithmetic conventions. They generally summarize effects for different target populations.
The key is to specify what a set-level effect represents and what units the aggregation is intended to represent. Giving every set equal weight describes an average across sets. Weighting by the number of relevant units describes an average across those units, provided the set-specific effects and weights refer to the same target units.
For causal interpretation, write \(Y_i(1)\) and \(Y_i(0)\) for unit \(i\)'s potential outcomes under treatment and control. A unit-level effect is \(Y_i(1)-Y_i(0)\). A set-specific effect \(\tau_j\) is the mean of these effects over the target units in set \(j\). In practice, the individual causal effects are not both observed for any one unit. Researchers therefore estimate set effects using observed outcomes and the matching design.
For example, a matched set may contain treated and control units, and its estimated contrast may be the treated mean outcome minus the control mean outcome. Treating that contrast as an estimate of a causal effect requires assumptions, not just successful matching. At a minimum, the treatment and control outcomes must be comparable within the matched set for the target being estimated; treatment and control observations must be available for that target; observed outcomes must equal the potential outcomes under the treatment received; and one unit's treatment must not change another unit's outcome. The target also needs to be clear: an effect among treated units is not automatically an effect among every unit in the set.
Two Aggregations, Two Target Summaries
With equal-set weighting, a randomly selected matched set receives the same importance regardless of its size. The target is the average of the set-specific effects across the \(J\) sets. This may be useful when the matched sets themselves are the units of substantive interest, or when the analysis deliberately wants each stratum to count equally.
With size weighting, a unit drawn at random from the represented target units is more likely to come from a large set. The weight for set \(j\) is \(n_j/\sum_k n_k\), so the target is the average unit-level effect across the units counted by \(n_j\). In an ATT analysis, for example, if each set-specific effect is an average effect among its treated units, the relevant size is the number of treated units in that set. Weighting instead by the total number of treated and control units would describe a different target and would not, by itself, turn the ATT into an all-unit effect.
Worked Example: Equal Sets Versus All Represented Units
Suppose three non-overlapping matched sets have estimated effects of 2, 6, and 10 points. For this illustration, assume the corresponding set sizes are 2, 4, and 6 target units. Positive effects mean higher outcomes under treatment. All values are hypothetical.
Equal-set summary. Give each of the three sets weight \(1/3\):
This describes the average effect across the three sets, not the average across their 12 target units.
Size-weighted summary. Weight each effect by the number of target units in its set:
A randomly selected target unit from these sets is represented by the 7.333-point summary. It exceeds the equal-set summary because the larger sets have the larger estimated effects. Neither number is universally preferable: the appropriate one depends on whether the scientific target is an average set or an average represented unit.
Why the Summaries Differ
The difference has a simple algebraic explanation. Define \(\bar n=J^{-1}\sum_j n_j\), \(\bar\tau=J^{-1}\sum_j\tau_j\), and the covariance across sets, using divisor \(J\), as \(\operatorname{Cov}_J(n,\tau)=J^{-1}\sum_j(n_j-\bar n)(\tau_j-\bar\tau)\). Then the size-weighted effect can be written as:
To see the last equality, expand \(J^{-1}\sum_j n_j\tau_j\) as \(\bar n\bar\tau+\operatorname{Cov}_J(n,\tau)\), and note that \(J^{-1}\sum_j n_j=\bar n\). Dividing by \(\bar n\) gives the expression. Thus the two summaries agree if all sets have the same size, or if set size and set effect have zero covariance. They can differ when larger sets tend to have systematically larger or smaller effects.
This identity describes the arithmetic relationship, not a causal explanation for why effects differ. A difference between summaries does not establish that set size causes effect variation. It does show that the reported aggregate depends on how the effects are distributed across sets of different sizes.
Worked Example: Checking the Weighting Difference
Use the three effects and sizes from the preceding example. The equal-set mean effect is \(\bar\tau=6\) points, and the mean set size is \(\bar n=(2+4+6)/3=4\) units. The covariance across sets is:
The identity therefore gives:
This agrees with the direct calculation. The covariance is positive because the largest set also has the largest effect; size weighting consequently raises the aggregate by 1.333 points compared with equal-set weighting. If the largest sets instead had smaller effects, the covariance could be negative and size weighting could lower the aggregate.
Choose the Weight That Matches the Target
The phrase “size-weighted” is incomplete unless the size being counted is stated. Matched sets can contain different numbers of treated and control units. In a variable-ratio design, those counts can vary across sets. The correct weight depends on the target population represented by each set-specific effect.
| Set-specific effect represents | Natural aggregation weight | Interpretation |
|---|---|---|
| One set-level quantity per matched set | Equal weight for each set | Average effect across matched sets |
| Treated units within each set | Number of treated units in the set | Average effect across represented treated units, when each is counted once |
| All units in each set, with an effect defined for that full target | Total number of target units in the set | Average effect across represented units in the full target |
The table is not a recipe for turning any set contrast into any desired estimand. The set-specific effect must first have a defensible interpretation for the units counted by its weight. For instance, a treated-minus-control contrast within each set may target treated units under an ATT design. Weighting that contrast by total set size instead of treated count changes the relative influence of sets, but it does not automatically identify the average effect for all treated and control units together.
Worked Example: Treated-Unit Weighting in Variable-Ratio Sets
Consider two hypothetical matched sets. Set A contains 2 treated and 2 control units, and its estimated effect among treated units is 3 points. Set B contains 6 treated and 3 control units, and its estimated effect among treated units is 7 points. Assume the set estimates target the treated units in their respective sets and the treated counts are the appropriate weights.
Giving the sets equal weight yields:
There are 8 treated units in all, so weighting by treated count yields:
This size-weighted summary represents the average effect among the 8 treated units, subject to the causal assumptions for the set-specific estimates. Weighting by total set size would give:
That is a different weighted summary. Because the set effects here are defined for treated units, the treated-count calculation is the direct treated-unit average. The total-set-size calculation gives the second set more weight partly because it has more controls; it should not be labelled the ATT without an additional justification.
Estimation, Diagnostics, and Alternatives
In an analysis, let \(\hat{\tau}_j\) be the estimated effect in set \(j\). The corresponding plug-in summaries are obtained by replacing \(\tau_j\) with \(\hat{\tau}_j\) in the formulas. These are descriptive aggregations of the set estimates. Their causal interpretation inherits the assumptions and design choices used to estimate each \(\tau_j\); aggregation cannot repair confounding within sets, lack of treatment or control observations for the target, or a mismatch between the effect and the units counted by the weights.
Before reporting a summary, check the matched-set membership and counts, state exactly which units define \(n_j\), and verify that each \(\hat{\tau}_j\) refers to the same kind of effect. A useful diagnostic is to compare the equal-set and chosen size-weighted summaries and inspect the set-level effects alongside set sizes. A large difference flags that weighting matters; it does not by itself indicate which summary is correct. Also check for reused controls: if a unit appears in multiple sets, counting every set appearance as a distinct person changes the represented target. The weighting must then reflect the intended unit or match-incidence target rather than silently counting duplicates as new units.
These summaries are alternatives, not competing estimators of one fixed estimand in every setting. An equal-set average is appropriate when the target is the typical matched set. A size-weighted average is appropriate when the target is a typical represented unit and the weights count those units. If the intended target is defined in another way, use weights that represent that target and state the choice. Inference must also account for the design and dependence structure; the point-estimate formulas alone do not determine a valid standard error.
Practical Decision Sequence
A transparent analysis can make the weighting decision explicit before calculating the aggregate:
Specify which units the effect in each matched set describes, such as treated units or all units in that set.
Decide whether the target is an average set or an average of a specified group of represented units.
Use equal weights for an average across sets, or count the relevant target units for a unit-average summary. Report the weight definition and denominator.
Compare the resulting summaries, inspect influential sets, and ensure matching assumptions and any reuse of units are handled consistently.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two matched sets have effects of 4 and 10 points and sizes of 3 and 9 target units. Calculate the equal-set and size-weighted summaries. Which set pulls the unit-weighted summary upward?
- When will equal-set and size-weighted effects necessarily be equal because of the set sizes, regardless of the set-specific effects?
- A set-specific ATT estimate represents treated units. Each set has 5 treated units, but the numbers of controls differ. What weights produce the average across represented treated units?
- Why does weighting an ATT set contrast by total set size not automatically estimate the average treatment effect among all units?
- In one set, the same control unit is reused in another set. What issue arises if both set appearances are counted as separate units in the size-weighted summary?
Solutions. (1) The equal-set summary is \((4+10)/2=7\) points. The size-weighted summary is \([3(4)+9(10)]/(3+9)=102/12=8.5\) points. The larger set, with a 10-point effect, pulls the unit-weighted summary upward. (2) If all sets have the same size, then each set receives the same normalized size weight, so both summaries equal the arithmetic mean of the set effects. (3) Give each set weight equal to its treated-unit count, here 5; because the counts are equal, this produces the equal-set numerical average as well. (4) The total count includes controls, while the set effects represent treated units; those weights do not define an average over all units' effects, and the contrast has not thereby been shown to identify that different target. (5) Counting appearances separately gives the reused control extra representation and may alter the target. Define whether the target counts distinct units or match incidences, then use weights consistent with that definition.