Two Targets, Two Questions
A matched analysis can describe a causal effect for the particular units that were retained in the realized data set. A different question asks about the average effect in a larger population from which those units might have been sampled. These targets are related, but they are not interchangeable: the first is defined by a finite set of units, while the second is defined by a population distribution and a rule for determining who belongs to the target group.
This distinction matters even when the matching procedure is fixed and the analysis is conducted correctly. A sample average can differ from its population counterpart simply because the realized retained units have a different mix of effect-modifying characteristics. The difference is about which units define the target, not necessarily about bias or a failure of the matching procedure.
The finite-sample estimand is a well-defined causal quantity even though, for each unit, only one of \(Y_i(1)\) and \(Y_i(0)\) is observed. Its definition uses both potential outcomes; an analysis must use design and assumptions to estimate it. It is not the same thing as the observed treated-minus-control difference, nor is it automatically the effect for everyone eligible for the study.
The rule \(r(X)\) is part of the definition of the population target. It might retain units in a specified covariate region or represent a fixed eligibility rule. If retention is determined by a matching algorithm that uses the whole realized sample, whether a unit is retained can depend on the other sampled units. In that situation, there is not automatically one fixed rule \(r(X)\); a superpopulation target must be defined for the sampling and retention procedure being considered. The simple results below use a fixed, unit-level rule based on pretreatment covariates.
When the Targets Connect
Suppose units are sampled independently from the source population, and a fixed rule \(R=r(X)\) retains each unit based on its covariates. Conditional on being retained, a sampled unit has the same covariate and potential-outcome distribution as a population unit conditional on \(R=1\). Consequently, if \(m\) units are retained, their average true effect has the superpopulation target as its conditional expectation. This is a repeated-sampling statement; it does not assert that the realized average equals the population mean.
To see why, condition on which \(m\) sampled units were retained. Under independent sampling and a fixed unit-level rule, each retained unit has the conditional distribution of a source-population unit given \(R=1\). Linearity of expectation then gives
This argument concerns the true effects among retained units, not the observed-data estimator. To identify those effects from treatment and outcome data, additional causal assumptions are needed. For example, consistency requires the observed outcome to equal the potential outcome under the treatment actually received. Conditional exchangeability requires \(Y(a)\) to be independent of treatment assignment given measured pretreatment covariates among the target units, for \(a=0,1\). Positivity requires a positive probability of each treatment level at covariate values represented in that target. Matching does not make these assumptions true by itself.
The repeated-sampling result also depends on how the sample and target are defined. A convenience sample may not represent the stated source population. A matching algorithm whose retention decisions depend on sample-wide competition may not behave like independent retention under a fixed \(r(X)\). And if the retained sample has no treatment overlap in part of its covariate distribution, a causal effect for all retained units may not be identified from the observed data without further modeling or a narrower target.
Worked Example: The Effect in One Realized Matched Sample
Consider a hypothetical matched sample with four retained units. Their potential outcomes, in outcome units, are shown below. Only one potential outcome per unit would be observed in an actual study, but both are listed here to define the finite-sample target.
| Retained unit | \(Y_i(1)\) | \(Y_i(0)\) | \(\tau_i=Y_i(1)-Y_i(0)\) |
|---|---|---|---|
| 1 | 8 | 6 | 2 |
| 2 | 9 | 5 | 4 |
| 3 | 5 | 5 | 0 |
| 4 | 7 | 5 | 2 |
There are \(m=4\) retained units. The finite-sample estimand is
Thus, for these four units, the average individual effect is 2 outcome units. This conclusion is specifically about the realized retained units. It does not establish that the population average for all units, or even the population average among units selected by a retention rule, is 2.
Worked Example: A Population Target and a Different Sample Average
Suppose a hypothetical source population has two covariate groups. Group A makes up 40% of the population and has an average treatment effect of 1 outcome unit. Group B makes up 60% and has an average treatment effect of 5 outcome units. A fixed pretreatment-covariate rule retains half of group A and one quarter of group B.
The probability that a population member is retained from group A is \(0.40(0.50)=0.20\). The corresponding probability for group B is \(0.60(0.25)=0.15\), so \(P(R=1)=0.20+0.15=0.35\). The target effect among retained population members is
Now suppose a hypothetical realized sample has four retained units: one from A and three from B. If their individual effects are 1, 5, 5, and 1 outcome units, the finite-sample retained effect is
The sample effect is 3, while the population target is approximately 2.714. The difference is consistent with sampling variation in the mix of retained groups. Under the stated independent-sampling and fixed-rule assumptions, the expected retained-sample effect over repeated samples is 2.714; this one realization need not equal it.
Diagnosing a Target Mismatch
Before interpreting a matched effect, specify whether the intended claim concerns the retained units actually observed or a broader population defined by a retention rule. Then check whether the assumptions connecting those targets are plausible. Useful diagnostics include:
- Sampling frame: Do sampled units represent the source population named in the population estimand, or are some groups systematically absent?
- Retention rule: Is the target group defined in advance by covariates, or does retention depend on competition among units in the realized sample?
- Covariate composition: Do retained sample units show an unusually large or small share of important effect modifiers compared with the intended population target?
- Outcome and treatment support: Are both treatment levels represented among the retained units across the covariate regions needed for the causal comparison?
These checks address distinct issues. Covariate balance between treated and control units within a matched sample is relevant to the credibility of a causal comparison, but it does not show that the retained units represent the source population. Conversely, a sample may reflect the intended target population's covariate mix while still having residual confounding or inadequate treatment overlap. Target definition, sampling, and causal identification should therefore be assessed separately.
| Quantity | What it averages over | Interpretation |
|---|---|---|
| Finite-sample retained effect | The retained units in the realized sample | Average causal effect for those particular units |
| Superpopulation retained-group effect | The population distribution conditional on the specified retention rule | Average causal effect for that population target group |
| Observed matched estimate | Observed outcomes and treatment assignments, using an analysis procedure | An estimate of a causal target only under suitable identification and analysis assumptions |
Worked Example: A Sample That Does Not Represent Its Stated Population
Suppose a hypothetical source population has two groups with average effects of 2 and 6 outcome units. The first group comprises 75% of the target population and the second 25%. The population effect is
Imagine, however, that the realized retained sample contains one unit from the first group and three from the second, with effects 2, 6, 6, and 6. Its finite-sample average is
The value 5 is the correct finite-sample average for these four units; it is not the population average of 3. If this group imbalance reflects a nonrepresentative sampling or retention process rather than ordinary random fluctuation under a suitable sampling design, the earlier repeated-sampling argument does not justify using the sample average as a population estimate. A population claim would require a defensible sampling model or additional adjustment assumptions. No causal estimator can, by its calculation alone, repair an incorrectly specified target or an unrepresentative sample.
Common Pitfall and Limits
A common mistake is to say, “The matched effect is the population effect because matching made the treatment groups comparable.” This conflates two questions. Comparability concerns whether the treated and control outcomes can identify an effect for the chosen target. Representativeness concerns whether that target corresponds to the population of interest. Matching may improve comparability on measured covariates while changing which units remain in the analysis. It does not, by itself, make the retained units representative of a broader population.
The finite-sample estimand is often the most direct target when the scientific question is about the actual retained units. A superpopulation estimand is appropriate when the question concerns a population beyond the observed sample, but it requires an explicit population definition and assumptions about sampling and retention. When the retention rule is data-adaptive or depends on the whole matched sample, the simple fixed-rule derivation above is insufficient; the population analogue must be defined to reflect that procedure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A matched sample retains six people. What does its finite-sample retained-unit estimand average over?
- State the key sampling and retention conditions used in the derivation connecting a retained-sample average effect to a population target.
- A realized retained-sample effect is 4 outcome units, while the specified population target is 3 outcome units. Does that difference alone disprove the repeated-sampling result?
- Why does balance between treated and control units not establish that retained units represent a source population?
Solutions. (1) It averages the six retained individuals' treatment effects, \(Y_i(1)-Y_i(0)\), and is a finite-sample target. (2) The argument assumes independent sampling from the source population and a fixed unit-level retention rule based on pretreatment covariates, with a positive probability of retention. (3) No. A realized average can differ from the population target; the result concerns its expectation over repeated samples under the stated assumptions. (4) Balance compares treated and control covariates within the retained analysis. Representativeness compares the retained units' distribution with the source population or specified population target, which is a different question.