Covariate Distributions as Target-Population Diagnostics

Why Look Beyond a Matching Summary?

A matching procedure can retain some units, discard others, or give units different weights. The resulting estimate therefore represents a particular distribution of observed covariates, not automatically the distribution of every unit in the original sample. Comparing covariate distributions before and after matching helps describe which kinds of units contribute to the matched target. A separate comparison of matched treated and control groups helps assess whether the groups are similar on measured covariates.

These comparisons answer related but distinct questions. A matched target may differ from the original sample because certain units are no longer represented; that is a target-composition shift. The matched treated and control groups may also differ from each other on a covariate; that is residual imbalance. Neither finding alone establishes whether the estimated effect is causally valid. Distribution diagnostics describe observed covariates, not unmeasured covariates or potential outcomes.

Definition: Let $X_i$ be a covariate for unit $i$ in an eligible sample of $n$ units. Given nonnegative target weights $w_i$ with $\sum_{i=1}^n w_i>0$, the weighted empirical distribution of a scalar covariate is $$ F_w(x)=\frac{\sum_{i=1}^n w_i I(X_i\leq x)}{\sum_{i=1}^n w_i}. $$ Here $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. The weights specify how much each unit contributes to the population being described. Equal weights describe the original sample; zero weights exclude units; and positive unequal weights give units different contributions.

In matching, $w_i$ must be chosen to correspond to the target population of interest. For example, an ATT-oriented target may be described using weights that represent the treated units, while another matching design may induce a different target. Do not combine treated and control weights indiscriminately: the correct weights depend on the estimand and the matching design. The earlier material on matching weights and discarded units provides the design context; here the focus is on reading the resulting covariate distributions.

What the Distributions Show

For a numeric covariate, useful summaries include the weighted mean, selected quantiles, and the weighted empirical CDF. The weighted mean is

\bar{X}_w=\frac{\sum_{i=1}^n w_iX_i}{\sum_{i=1}^n w_i}.

The CDF at $x$ is the weighted share of units with covariate value at or below $x$. Comparing CDFs across populations shows where one distribution accumulates more mass than another. Quantiles, ranges, and plots can further reveal shifts in location, spread, tails, or multimodality. For a categorical covariate, compare category proportions directly. If $C$ is a specified category or set of categories, its weighted proportion is

\hat{P}_w(X\in C)=\frac{\sum_{i=1}^n w_i I(X_i\in C)}{\sum_{i=1}^n w_i}.

This indicator must refer to the category of interest. In particular, for a binary covariate coded 0 and 1, $F_w(1)=1$, because every observation is at most 1. The proportion coded 1 is instead $\sum_i w_i I(X_i=1)/\sum_i w_i$. More generally, a CDF gives proportions for cumulative sets, not for every arbitrary category.

Worked Example: Identifying an Age Shift

Suppose the candidate target sample has ten units with ages, in years, $25, 28, 30, 32, 35, 38, 40, 42, 45,$ and $50$. After matching, six of these units receive target weight 1 and the other four receive weight 0. The retained ages are $25, 28, 30, 32, 35,$ and $38$. What does this say about the represented age distribution?

Original mean. With equal original weights, $\bar{X}=(25+28+30+32+35+38+40+42+45+50)/10=365/10=36.5$ years.

Matched-target mean. The positive weights sum to 6, so $\bar{X}_w=(25+28+30+32+35+38)/6=188/6=31.33$ years.

The matched target's mean age is 5.17 years lower. The distribution comparison gives additional detail: the original share aged at most 35 is $5/10=0.500$, whereas the matched-target share is $5/6=0.833$. The matched target is more concentrated among younger units in this example. This is evidence about the composition of the represented sample, not evidence that age caused units to be retained or that the matched effect applies to every original unit.

A concise diagnostic often compares the original target distribution with the postmatching target distribution and, separately, compares covariate distributions across matched treatment groups. These should be reported as separate comparisons. For example, if matching targets the treated units, the treated units' covariate distribution describes the target composition; the matched controls should approximate that composition for an ATT interpretation. If the matched controls still have a different distribution, that is a balance concern, not a reason to redefine the target silently.

Worked Example: Weighted Proportions for a Categorical Covariate

Suppose the target weights for 30 matched units sum to 30. Their weighted contributions by region are 16 for North, 12 for Central, and 2 for South. In the original eligible sample, the corresponding proportions were 0.40, 0.35, and 0.25. Calculate and interpret the postmatching proportions.

For North, $ \hat{P}_w(X\in\{\text{North}\})=16/30=0.5333$. For Central, the proportion is $12/30=0.4000$, and for South it is $2/30=0.06667$. These sum to $30/30=1$, as category proportions for an exhaustive, mutually exclusive classification should.

Compared with the original sample, the matched target gives more weight to North (53.33% rather than 40.00%) and less to South (6.667% rather than 25.00%). The calculation uses the indicator for each named category. Coding North, Central, and South as 1, 2, and 3 and reading a CDF at 3 would give 1, not the South proportion; the category-specific indicator is essential.

Distribution Comparisons and Their Limits

A difference in means is easy to communicate but cannot describe an entire distribution. Two groups can have identical means and still differ in spread, skewness, tails, or clusters. A standardized mean difference (SMD) summarizes a difference in means relative to a standard-deviation scale; it is useful for comparing the magnitude of mean differences across covariates measured in different units. It does not measure all forms of distributional difference. The standard deviation used for scaling should be stated and chosen consistently, especially when comparing before and after matching.

For a scalar covariate, one possible overall comparison is the largest absolute difference between two weighted empirical CDFs, $\sup_x|F_1(x)-F_2(x)|$. This emphasizes the largest cumulative discrepancy, but it does not identify its substantive cause and can be sensitive to sample size and discreteness. Plots, quantiles, and subject-matter interpretation remain important. For multiple covariates, marginal comparisons alone can also miss differences in their joint distribution: two groups can have similar age and disease-severity margins but different age-by-severity combinations.

Worked Example: Equal Means Can Hide Different Distributions

Consider a baseline biomarker measured in arbitrary units among four matched treated units, with values $0, 0, 10, 10$, and four matched controls, with values $4, 4, 6, 6$. Compare the means and empirical CDFs.

The treated mean is $(0+0+10+10)/4=5$ units. The control mean is $(4+4+6+6)/4=5$ units. Thus the mean difference is $5-5=0$ units; a mean-based diagnostic alone would show no difference.

At $x=0$, the treated CDF is $2/4=0.5$, while the control CDF is $0/4=0$, a difference of 0.5. At $x=4$, both CDFs equal 0.5. At $x=6$, the treated CDF remains 0.5 and the control CDF is $4/4=1$, a difference of 0.5. The groups have equal means but substantially different distributions: treated units are at the extremes, while controls are concentrated in the middle. This is residual distributional imbalance on the measured biomarker, despite a zero difference in means.

A short justification explains why the weighted category formula represents a proportion. Each term $w_iI(X_i\in C)$ contributes the unit's weight if that unit belongs to $C$, and contributes zero otherwise. Summing therefore gives the total weight in $C$; dividing by the positive total weight across all units gives the share of represented weight in that category. Nonnegative weights ensure these shares are between 0 and 1, and the positive-denominator condition makes them defined.

How to Diagnose Target Representation

Specify the population being described.
State whether the comparison concerns the original eligible sample, the matched target, or one treatment group within the matched sample. Name the weights that define each distribution.

Compare the target composition before and after matching.
For numeric covariates, inspect means and distributional summaries such as quantiles and CDFs. For categorical covariates, calculate the weighted proportion in each relevant category.

Assess treated-control similarity within the matched design.
Compare the covariate distributions using the weights appropriate to each group and the intended target. Do not confuse similarity between groups with similarity to the original sample.

Interpret changes substantively.
Describe which covariate values or categories became more or less represented, whether meaningful treated-control differences remain, and which units or settings the result may not describe.

In practice, use more than one diagnostic. Tables can report original and matched means, standard deviations, category proportions, and SMDs; distribution plots can expose differences that these summaries miss. For highly skewed covariates, show quantiles or empirical CDFs rather than relying on a mean alone. For categorical covariates, show all important category proportions rather than a single coded-number summary.

These are descriptive design diagnostics, not hypothesis tests that certify a design as good. A nonsignificant balance test can occur simply because a sample is small, and a significant test can reflect a small, practically unimportant difference in a large sample. The question is the size and relevance of remaining differences, not whether a test crosses a threshold. Diagnostics should be computed using the actual matching weights and should clearly identify whether weights apply to matched sets, treated units, controls, or another target.

Common Pitfalls and Alternatives

Calling target shift imbalance. If matching changes which units contribute, the target's distribution may differ from the original population even when treated and control distributions are similar within the matched target. Report these as separate facts.
Using only means or SMDs. Equal means do not imply equal distributions. Add quantiles, plots, or CDF comparisons when tails or nonlinear patterns matter.
Using a CDF value as an arbitrary category proportion. $F_w(x)$ measures the weighted share at or below $x$. Use $I(X_i\in C)$ to calculate the share in a specified category $C$.
Ignoring the weights or their denominator. A weighted distribution requires nonnegative weights with positive total weight. The ordinary unweighted matched sample can describe a different population when weights vary.
Assuming good observed balance proves causal validity. Similar measured covariates do not establish balance on unmeasured variables, correct outcome measurement, or the assumptions needed for a causal interpretation.

The main alternative to a distribution-based review is a compact set of mean differences, SMDs, and variance ratios. These summaries are easy to compare across many covariates, but can conceal distributional shape. Conversely, plots and full CDF comparisons are richer but can be difficult to summarize across many variables. A practical report uses concise numerical summaries plus targeted plots for covariates with skew, outliers, important categories, or substantive importance. No diagnostic replaces a clear statement of the matched target.

Key takeaway: Covariate distributions describe who is represented after matching and whether measured covariates remain distributed similarly across matched groups. Use weights tied to the estimand, compare more than means, and keep target-composition change distinct from residual imbalance.

Check Your Understanding

Answer each question, then compare with the solutions below.

Why must the total weight be positive when defining a weighted empirical distribution?
A binary covariate is coded 0 or 1. Which expression gives the weighted proportion coded 1, and why does $F_w(1)$ not give it?
Matching changes a target's mean age from 42 years to 35 years, while the matched treated and control age distributions are similar. Is this necessarily a contradiction? Explain.
What limitation remains if two matched groups have identical means for every measured covariate?

Solutions. (1) The total weight is the denominator; if it is zero, the weighted distribution is undefined. (2) Use $\sum_i w_i I(X_i=1)/\sum_i w_i$. Since every coded value is at most 1, $F_w(1)=1$ when total weight is positive. (3) No. The target may have shifted toward younger units compared with the original sample even though treated and control units are similar within that new target. Those are distinct comparisons. (4) The groups may still differ in variances, tails, categories, nonlinear or joint covariate patterns, and unmeasured characteristics; equal means alone establish none of those forms of similarity.