Distributional Balance With Empirical Cdfs

Why Compare Whole Distributions?

A weighted mean is a useful summary of covariate balance, but it compresses an entire distribution to one number. Two treatment groups can have the same weighted mean while differing in spread, skewness, tails, or the locations of point masses. Weighted empirical cumulative distribution functions (CDFs) and quantile plots provide direct ways to inspect such differences. They complement moment-based checks; they do not replace checks of positivity, weight behavior, or other aspects of an analysis.

For a baseline covariate $X$, the comparison is between its distributions in the treatment groups after applying the analysis weights. The interpretation is in terms of the weighted pseudo-populations represented by those weights, not necessarily the raw groups.

Weighted Empirical CDFs

Definition: For treatment group $a\in\{0,1\}$, let $I_a=\{i:A_i=a\}$, let $w_i\geq 0$ be the analysis weight for individual $i$, and suppose $W_a=\sum_{i\in I_a}w_i>0$. The normalized weighted empirical CDF of covariate $X$ in group $a$ is $$ \widehat F_{a,w}(x)=\frac{\sum_{i\in I_a}w_i\,\mathbf{1}(X_i\leq x)}{W_a}. $$ It is a nondecreasing, right-continuous step function with limits 0 and 1 at the two ends of the real line. Its jump at a distinct observed value is the total weight at that value divided by $W_a$.

Normalizing within each group is essential: it makes each weighted distribution have total probability one. Multiplying every weight in one group by the same positive constant leaves that group's CDF unchanged. The empirical CDF can therefore be read as the proportion of total group weight at or below $x$.

A direct numerical summary of the gap is $D_w=\sup_x|\widehat F_{1,w}(x)-\widehat F_{0,w}(x)|$. For empirical CDFs the supremum is attained at one of the observed covariate values. The maximum gap reports the largest vertical separation; inspecting the curves as well shows where the separation occurs and whether one group tends to have larger values.

Worked Example: Equal Means, Different Distributions

Suppose the weighted distribution in group 1 places weight 1 at $X=0$ and weight 1 at $X=2$. In group 0, it places weight 2 at $X=1$. Each group has total weight 2. Calculate the means and compare the CDFs.

The weighted means are $ (1(0)+1(2))/2=1 $ in group 1 and $2(1)/2=1$ in group 0. Thus the mean difference is zero.

The group 1 CDF is 0 below 0, $1/2$ from 0 up to (but not including) 2, and 1 at and above 2. The group 0 CDF is 0 below 1 and 1 at and above 1. At $x=0$, the CDF values are $1/2$ and 0. At $x=1$, they are $1/2$ and 1. At $x=2$, both are 1. Consequently, $D_w=1/2$, despite the identical means.

The groups differ sharply: group 1 has values on either side of 1, whereas all the group 0 weight is at 1. A mean-only check would miss that distinction.

Quantile Plots and Their Interpretation

Definition: For $0<u\leq 1$, the weighted empirical quantile in group $a$ is $$ \widehat Q_{a,w}(u)=\inf\{x:\widehat F_{a,w}(x)\geq u\}. $$ A quantile comparison plots $\widehat Q_{1,w}(u)$ against $\widehat Q_{0,w}(u)$ over selected probability levels $u$. Equality of the two quantiles at a level means the groups have the same weighted value at that part of their distributions.

Because these are step functions, empirical quantiles can repeat across ranges of $u$, especially when weights are concentrated or there are ties. The inverse-CDF definition above fixes the convention: it chooses the smallest observed value at which the CDF reaches the requested probability. Software may use other interpolation conventions, so the convention should be stated when exact numeric quantiles matter.

Worked Example: Reading Weighted Quantile Differences

In group 1, $X=1,2,4$ have weights $1,2,1$. In group 0, $X=1,3,4$ have weights $1,1,2$. Both groups have total weight 4. Find the CDF values and quantiles at $u=0.25,0.50,0.75$.

For group 1, the cumulative weights at $1,2,4$ are $1,3,4$, giving CDF values $0.25,0.75,1$. Therefore its quantiles at the specified levels are $1,2,2$.

For group 0, the cumulative weights at $1,3,4$ are $1,2,4$, giving CDF values $0.25,0.50,1$. Its quantiles are $1,3,4$. The group 1 minus group 0 quantile differences are thus $0,-1,-2$ at the three levels. The groups agree at the lower quartile but differ at the median and upper quartile.

The CDF gap also identifies a discrepancy: at $x=2$, the CDFs are $0.75$ and $0.25$, so the gap there is $0.50$. This calculation and the quantile comparison describe the same distributions from two complementary viewpoints.

Two Properties That Clarify the Diagnostics

Result: Two probability distributions on the real line are equal if and only if their CDFs agree at every real $x$. Thus, zero maximum CDF gap means exact equality of the two weighted empirical distributions, not just agreement of selected summaries.

Justification. If the distributions are equal, their probabilities of every set $(-\infty,x]$ agree, so their CDFs agree. Conversely, if their CDFs agree everywhere, then their probabilities of all half-lines $(-\infty,x]$ agree. These half-lines form a collection closed under finite intersections and generate the Borel sets on the real line. The uniqueness theorem for probability measures on a generating collection then implies that the two distributions agree on every Borel set. Applied to the normalized weighted empirical distributions, this proves the claim. $\square$

The maximum gap also bounds discrepancies in smooth summaries, under explicit conditions. This makes precise why examining a CDF can provide more information than checking a particular mean.

Proof. For any $x\in[l,r]$, absolute continuity gives $f(x)=f(l)+\int_l^r f'(t)\mathbf{1}(x>t)\,dt$. Taking expectations under distribution $j$, for $j\in\{0,1\}$, and interchanging expectation and integration is justified because $\int_l^r |f'(t)|\,dt<\infty$ and the indicator is bounded by 1. Since $P_j(X>t)=1-F_j(t)$, subtraction yields $$ E_1[f(X)]-E_0[f(X)] =\int_l^r f'(t)\{F_0(t)-F_1(t)\}\,dt. $$ Taking absolute values, using $|F_0(t)-F_1(t)|\leq D$, and integrating gives the stated bound. $\square$

For $f(x)=x$, the bound applies to the mean difference when the support is bounded. But a small or zero mean difference does not imply a small $D$: positive and negative contributions can cancel in the signed integral. The first worked example exhibits that cancellation.

Using the Plots Carefully

A useful diagnostic sequence is to calculate normalized weighted CDFs, inspect their overlaid step curves, report a maximum gap if a single-number summary is helpful, and compare weighted quantiles at interpretable probability levels. The maximum gap is sensitive to the largest local discrepancy; a quantile plot makes the direction and location of differences easier to describe. Neither display alone identifies why the weights produced the observed balance pattern.

Look across the range. A CDF gap concentrated in one tail is different from a persistent shift over most of the distribution, even if the maximum gaps are similar.
Check the weights and support. A weighted CDF is only as informative as the observations and weights supporting it. A sparse tail or a few highly influential weights can make the apparent distribution unstable; inspect weight distributions, effective sample size, and covariate-specific support alongside the plot.
Respect the variable type. For an ordered categorical variable, a CDF depends on the meaningful category order. For an unordered categorical variable, an arbitrary numerical ordering makes a CDF gap misleading; compare weighted category proportions instead.
Keep the scope clear. Marginal CDFs compare one covariate at a time. Agreement for separate covariates does not establish agreement in their joint distribution or prove that confounding has been removed.
Do not mistake a diagnostic for a test. The empirical maximum gap is descriptive. Treating it as a formal two-sample test statistic requires an appropriate reference distribution; estimated weights and dependence in longitudinal data can invalidate a naive unweighted reference calculation.

Compared with higher-moment balance checks, the CDF uses the ordering of all observed values and can reveal differences that a finite list of moments may overlook. Compared with standardized mean differences, it does not reduce the comparison to a mean scaled by a chosen measure of dispersion. Its costs are that plots can be harder to summarize across many covariates and that the maximum gap does not convey every feature of the discrepancy. These methods are complementary: no single balance summary establishes that a causal analysis is correctly specified.

Worked Example: Computing a Weighted CDF Gap

Suppose group 1 has $X=0,1,3$ with weights $2,1,1$, and group 0 has $X=0,2,3$ with weights $1,2,1$. Calculate each CDF at the observed values and find the maximum gap.

Both total weights are 4. At $x=0$, the CDFs are $2/4=0.50$ and $1/4=0.25$, so the gap is $0.25$. At $x=1$, they are $3/4=0.75$ and $1/4=0.25$, so the gap is $0.50$. At $x=2$, they are $0.75$ and $3/4=0.75$, so the gap is 0. At $x=3$, both are 1, so the gap is 0. Below 0 both CDFs are 0; above 3 both are 1. Therefore $D_w=0.50$.

If every group 1 weight were multiplied by 10, its total would become 40 and each numerator would also be multiplied by 10. For example, its CDF at 1 would remain $30/40=0.75$. The within-group normalization is what makes the comparison independent of a common weight scale.

Common Pitfalls

Equating equal means with balance. The mistaken reasoning is that if the weighted means match, the covariates must be balanced. The correction is that means can agree while distributions differ; inspect CDFs or quantiles as well.
Comparing unnormalized cumulative weights. Raw cumulative totals depend on group size and weight scale. Divide by each group’s total weight before comparing distributions.
Reading the maximum gap as the whole story. The same maximum can occur in different regions and with different patterns. Inspect the curves and report where the gap occurs.
Interpreting marginal agreement as joint balance. Matching each covariate separately does not guarantee that combinations of covariates are similarly distributed.

Check Your Understanding

Answer each question, then compare with the solutions below.

What is the weighted empirical CDF at $x$ if a group has total weight 10 and weight 7 among observations with $X_i\leq x$?
Two weighted distributions have CDFs that differ by 0.3 at some value. What does this establish about their maximum CDF gap?
Can two groups have the same weighted mean and different weighted CDFs? Give a numerical example.
Why is an empirical CDF gap generally not an appropriate comparison for an unordered categorical covariate?

Solutions. (1) It is $7/10=0.7$, by dividing cumulative weight by total group weight. (2) The maximum gap is at least 0.3, since it is the supremum over all values and includes the value in question. (3) Yes: one group with equal weight at 0 and 2 has mean 1, while a second with all weight at 1 also has mean 1; their CDFs differ by 0.5. (4) The CDF relies on an ordering of values. An arbitrary order can change the gaps, so weighted category proportions are the appropriate direct comparison.