Empirical Distribution and Kolmogorov-Smirnov Diagnostics

Why Compare Entire Distributions?

Balance diagnostics should show whether the treatment and comparison groups have similar covariate distributions after the chosen matching or weighting procedure. A difference in means can miss important features: two groups may have the same mean while differing in spread, skewness, or tails. Quantile-quantile plots help reveal such patterns; empirical cumulative distribution functions (ECDFs) provide another direct comparison, and the Kolmogorov–Smirnov (KS) distance reduces that comparison to one number.

The KS distance is a descriptive summary of the largest gap between the groups’ cumulative distributions. It answers a specific question: across all possible cut points on the covariate scale, what is the greatest difference in the proportions of the two groups at or below a cut point? It can flag distributional imbalance that a mean-based summary does not show. It does not, by itself, determine whether matching or weighting succeeded for every covariate or establish that a causal estimate is unbiased.

Empirical Cumulative Distribution Functions

In a balance assessment, calculate one ECDF for the treated group and one for the comparison group using the same covariate and the same measurement scale. Each curve gives the fraction of that group with covariate values no greater than \(x\). For example, if the treated ECDF at \(x=10\) is 0.70, then 70% of the treated observations have values at or below 10.

The ECDF is right-continuous: at an observed value, it includes all observations tied at that value. Ties therefore produce jumps larger than \(1/n\). This is not a defect; it reflects the observed data. Although the covariate of interest is often continuous in principle, recorded measurements may have ties because of rounding or a discrete measurement scale.

The Kolmogorov–Smirnov Distance

For any cut point \(x\), the signed difference \(\widehat{F}_T(x)-\widehat{F}_C(x)\) compares the treated and comparison proportions at or below \(x\). The absolute value measures the size of that discrepancy without regard to its direction. The largest absolute difference over all cut points is \(D\). For finite samples, the maximum can be found by checking the observed covariate values in both groups, including tied values. Between consecutive distinct observed values, neither ECDF changes, so their difference is constant.

The distance is expressed on a proportion scale, not in the covariate’s measurement units. If \(D=0.20\), then at the cut point where the maximum occurs, the groups’ cumulative proportions differ by 20 percentage points. That does not mean every cut point differs by 20 points, nor does it say which group has larger covariate values overall. To understand direction and location, inspect the signed difference and the ECDF curves as well as \(D\).

Justification and a Second Distribution Pattern

The calculation follows directly from what an ECDF represents. At each \(x\), \(\widehat{F}_T(x)\) and \(\widehat{F}_C(x)\) are the respective fractions at or below that cut point. Their difference is therefore the difference in cumulative fractions at \(x\). Taking the absolute value removes the sign, and taking the supremum selects the greatest discrepancy across the covariate scale. In finite data, each curve changes only at observed values, so comparing the curves at the pooled set of observed values is sufficient to obtain the maximum.

Weighted ECDFs After Weighting

When a balance assessment concerns a weighted sample, an unweighted ECDF describes the observed sample rather than the weighted target distribution. For nonnegative weights \(w_i\) with positive total weight, the weighted ECDF is the weighted fraction at or below \(x\):

Calculate this ECDF separately for each group, normalizing by that group’s total weight. Then use the same supremum-of-absolute-differences definition for the weighted KS distance. The weights must be nonnegative for this expression to represent a cumulative distribution. The covariate and its units must be the same in both groups, and the weights should be those defining the weighted comparison being diagnosed.

Reading the Diagnostic and Recognizing Its Limits

The KS distance is useful because it assesses a distributional feature across the whole covariate scale rather than at only one summary, such as the mean. A large distance can reveal a localized gap, a broad shift, or a tail difference. The distance does not identify which pattern produced it. Plotting both ECDFs helps locate where they separate; the signed difference helps identify which group has the larger cumulative proportion there. A smaller ECDF at a given \(x\) means a smaller fraction of that group is at or below \(x\), not automatically that the entire group is larger at every point.

Treat \(D\) as a descriptive balance diagnostic, not as an automatic pass/fail rule. Its interpretation depends on the covariate, study context, and intended analysis. A distance that appears modest may still matter if it occurs in a consequential region of the covariate distribution. Conversely, the distance does not communicate every feature of the distribution. It can also be affected by sparse observations and ties, especially in small samples.

The calculation does not require independent sampling as a condition for the descriptive comparison itself: it is computed from the two empirical distributions at hand. However, an inferential KS test and its p-value answer a different question and require assumptions appropriate to that inferential procedure. After matching or weighting, dependence induced by the design and the construction of the weights make an off-the-shelf two-sample p-value an inappropriate substitute for balance assessment. A small p-value is not a measure of practical imbalance, and a large one does not establish balance.

The ECDF comparison is univariate: it evaluates one covariate at a time. A set of small distances does not establish that the joint distribution of all covariates is balanced, and a distance for one covariate cannot diagnose imbalance in interactions or subgroups. It also cannot verify that the matching or weighting procedure made the right causal assumptions. Interpret it alongside other relevant diagnostics rather than treating a single number as a guarantee.

How It Compares With Other Balance Summaries

These diagnostics address related but nonidentical features. For example, quantile-quantile plots compare values at corresponding quantiles, while the KS distance compares cumulative proportions at common cut points. The ECDF display and its distance complement rather than replace mean differences, variance ratios, and quantile-quantile plots.

Common Pitfalls

• Using only the mean. Equal means do not imply equal distributions. Compare ECDFs when the aim includes checking distributional shape across the covariate range.
• Reporting a distance without its meaning. State that \(D\) is the maximum absolute difference in cumulative proportions, and give its proportion-scale interpretation.
• Ignoring the curves’ direction. Because \(D\) is absolute, it does not tell which group has the larger cumulative proportion at the maximizing cut point. Inspect the signed difference or both ECDFs.
• Dropping ties or using the wrong denominator. Include every observation at or below the cut point and divide by the group sample size, or by the group’s total weight for a weighted ECDF.
• Treating a KS test p-value as a balance score. A descriptive distance and an inferential test are not interchangeable. Balance assessment focuses on the observed distributions, not a test of whether a population-level null can be rejected.