Identifying the Population After Matching or Weighting

Why the Analysis Population Needs Its Own Description

After matching or weighting, the people contributing to an estimate may no longer represent the original eligible cohort in the same way. Matching can exclude individuals who lack suitable counterparts. Weighting can leave records in the data but give them different influence in the analysis. In either case, describing only the original cohort leaves readers without a clear account of which population the adjusted comparison represents.

The description should connect three things: the target estimand, the procedure used to construct the analysis sample or weights, and the characteristics of the resulting population. A matched estimate based on a subset should not be described as if every eligible person contributed. A weighted estimate should not be described as if its weighted population were simply the unweighted set of records.

These labels are related but not interchangeable. With matching, the analysis population consists of retained individuals, but the target may be described in terms of the treated individuals retained, the matched treated and comparison groups, or another explicitly defined target. With weighting, records may remain in the data while their contributions change; the resulting weighted distribution is often best described as a weighted or pseudo-population, not as a newly observed group of people.

Describe the Population Created by the Design

For matching, report the numbers eligible before matching and the numbers retained in each treatment group afterward. State whether treated individuals were unmatched or excluded, whether comparison individuals were reused, and whether matching was with or without replacement. The reporting should follow the actual design rather than assume a standard one-to-one structure. If matching used variable ratios, the count of retained individuals and the matching structure both help readers understand who contributes.

For weighting, report the number of individuals with nonzero weights in each group, the weight definition, and any trimming or truncation that changes which observations contribute or how strongly. A record with a very small positive weight technically contributes, but may have little influence; a large weight may have substantial influence. Reporting only the number of records is therefore insufficient to describe weighted representation.

In both cases, give the characteristics that help readers understand how the analysis population differs from the original cohort. Commonly useful characteristics include age, sex or gender as recorded, baseline disease severity, comorbidities, and other prespecified covariates relevant to the treatment decision or outcome. Report group-specific values, with denominators or weights made clear. The earlier material on covariate balance diagnostics addresses whether measured covariates are balanced; the task here is also to explain whose characteristics are being summarized and how the adjusted analysis population relates to the cohort.

Weighted Characteristics and Their Interpretation

A weighted mean of a covariate \(X\) in one treatment group is the sum of that covariate multiplied by each individual's analysis weight, divided by the sum of those weights in that group. For individuals indexed by \(i\), with weight \(w_i\), the weighted mean is

For a binary characteristic coded 1 when present and 0 otherwise, the same calculation gives the weighted proportion. The denominator is the sum of weights, not necessarily the number of records. It is therefore important to label a result as weighted and to state the treatment group and characteristic it describes.

For example, with the average treatment effect (ATE) as the target, inverse-probability weights are commonly defined as \(1/e(X_i)\) for treated individuals and \(1/[1-e(X_i)]\) for comparison individuals, where \(e(X_i)=P(A_i=1\mid X_i)\) is the propensity score. Under the usual conditions for this weighting approach, the weighted groups are intended to represent the eligible cohort's covariate distribution. With the average treatment effect on the treated (ATT) as the target, treated individuals commonly receive weight 1 and comparison individuals receive \(e(X_i)/[1-e(X_i)]\). The comparison group is then reweighted toward the treated group's covariate distribution.

This distinction can be justified using Bayes' rule. Let \(f(x\mid A=1)\) and \(f(x\mid A=0)\) denote covariate densities in the treated and comparison groups, and let \(P(A=1)=\pi\). Then

The right side is proportional to the treated covariate density, so normalizing the ATT-weighted comparison distribution gives the treated covariate distribution, in the population under the stated model and positivity conditions. In a finite sample, the weighted distributions need not match perfectly. This is why reports should give observed weighted summaries and diagnostics rather than assert that the distributions are identical by definition.

Different weights represent different target distributions. For example, overlap weights give treated individuals weight \(1-e(X_i)\) and comparison individuals weight \(e(X_i)\), emphasizing individuals with covariates associated with a meaningful chance of receiving either treatment. A report should identify the target estimand and weighting scheme rather than describe every weighted analysis as representing the entire original cohort.

Compare the Analysis Population With the Original Cohort

A clear account contrasts original-cohort and post-adjustment characteristics, without implying that the two populations are interchangeable. For each important characteristic, report values in the original eligible cohort and the matched or weighted analysis population. For matching, show the retained groups and make the unmatched exclusions visible. For weighting, provide unweighted cohort summaries alongside weighted summaries when that comparison helps explain how reweighting changes representation.

For weighted summaries, state whether weights were normalized and how. Multiplying every weight in a group by the same positive constant does not change its weighted mean or proportion, but it does change the sum of weights. Thus, when reporting a weighted total or an effective sample size, the normalization convention matters. When reporting a weighted mean, identify it as a weighted mean and give the corresponding group and target.

Do not describe the weighted population as a literal set of additional or duplicated people unless the procedure actually creates replicated records. A weighted pseudo-population is a representation induced by the weights. Nor should one assume that all original-cohort members contribute equally simply because they remain in the analytic file.

Diagnostics, Limitations, and Common Mistakes

Several warning signs should prompt careful reporting. Large losses from one treatment group indicate that the adjusted estimate may apply to a narrower population than the eligible cohort. Very large weights suggest that a few observations may dominate weighted summaries. Substantial differences in key covariates after adjustment indicate that the realized analysis population may not have the intended balance. Sparse overlap or positivity concerns can also limit which individuals are represented reliably. Report the relevant exclusions, weight distribution, and balance results; do not use a population label to conceal these features.

The description of an analysis population does not establish causal validity. A matched or weighted population can be described accurately while causal interpretation remains limited by confounding, model misspecification, measurement error, or other design problems. Causal interpretation requires the relevant assumptions, including conditional exchangeability and positivity for the target population, along with consistency and appropriate implementation. The population description tells readers who the estimate concerns; it does not prove those assumptions.

• Mistake: “The study included 200 patients, so all 200 contributed equally to the adjusted estimate.” Correction: report the retained counts for matching or the weights and weighted contribution for weighting.
• Mistake: “The matched estimate represents all eligible treated patients.” Correction: state how many treated patients were retained and identify the target implied by the estimand. A subset of treated patients may be represented if others could not be matched.
• Mistake: “The weighted sample contains 5.25 people because its weights sum to 5.25.” Correction: weight sums are not ordinary head counts. Report the number of records and distinguish it from weighted summaries or other weight-based quantities.
• Mistake: “The intended target distribution was achieved, so no weighted characteristics need to be reported.” Correction: show the observed weighted summaries and diagnostics. Finite-sample balance is not guaranteed by the intended weighting target.

A concise but complete report identifies the estimand, gives original and post-adjustment counts, describes the matching or weighting scheme, and summarizes important covariates in the analysis population. This lets readers distinguish a change in statistical adjustment from a change in who is represented.