Distance Metrics For Comparing Population Distributions

Why Compare Distributions With More Than One Measure?

A generated virtual population can match a target mean and variance yet still differ in its tails, shape, or location of individual values. Conversely, one summary may flag a discrepancy without indicating whether it is a small shift or a major mismatch. Distribution-comparison measures make these differences more explicit. They help evaluate how closely the generated population represents the intended target, but they do not by themselves establish that a population is scientifically adequate.

The earlier material on matching means, variances, and quantiles provides familiar summaries to compare. Here, the goal is to turn such comparisons into clearly defined errors and to add two measures that use the full univariate empirical distributions: Kolmogorov–Smirnov distance and Wasserstein distance.

Define the Distributions and the Comparison

Let $T$ denote the target distribution and $G$ the generated distribution for the same variable, measured on the same scale. For a finite virtual population with equal weights, treat each set of values as an empirical distribution: the probability assigned to any value is the fraction of population members having that value. For weighted populations, use normalized weights as probabilities in both distributions. A valid comparison requires the same variable, units, and interpretation in the target and generated populations.

Definition: A standardized moment error is a dimensionless difference between a chosen moment summary for $G$ and the corresponding summary for $T$, divided by a stated reference scale. One useful reporting convention is the error vector $ (E_\mu,E_{\sigma^2},E_{\gamma_1},E_{\gamma_2}) $, where $$ E_\mu=\frac{|\mu_G-\mu_T|}{\sigma_T},\qquad E_{\sigma^2}=\frac{|\sigma_G^2-\sigma_T^2|}{\sigma_T^2},\qquad E_{\gamma_1}=|\gamma_{1,G}-\gamma_{1,T}|,\qquad E_{\gamma_2}=|\gamma_{2,G}-\gamma_{2,T}|. $$ Here $\mu$ is the mean, $\sigma^2$ the variance, $\gamma_1=\mu_3/\sigma^3$ the skewness, and $\gamma_2=\mu_4/\sigma^4-3$ the excess kurtosis; $\mu_r$ is the $r$th central moment. The subscripts $T$ and $G$ identify target and generated distributions. This convention standardizes mean error by the target standard deviation, variance error by the target variance, and skewness and excess-kurtosis errors by using their dimensionless forms.

The vector reports different features separately rather than hiding them in a single score. A mean error of 0.2 means the generated and target means differ by one-fifth of the target standard deviation. A relative variance error of 0.2 means the variances differ by 20% of the target variance. Skewness and excess kurtosis errors are absolute differences between dimensionless shape summaries. If $\sigma_T=0$, the first two ratios are undefined; report absolute differences instead, and explain the degenerate target rather than inserting an arbitrary denominator.

Worked Example: Standardized Moment Errors

Suppose a hypothetical target population has values $0,0,2,2$, and a generated population has values $0,1,1,2$. Use equal weights and moments calculated over each finite population, dividing by its size.

Target moments. The target mean is $\mu_T=(0+0+2+2)/4=1$. Its deviations from the mean are $-1,-1,1,1$, so $\sigma_T^2=(1+1+1+1)/4=1$ and $\sigma_T=1$. Its third central moment is 0, giving $\gamma_{1,T}=0$. Its fourth central moment is 1, so $\gamma_{2,T}=1/1^2-3=-2$.

Generated moments. The generated mean is $\mu_G=(0+1+1+2)/4=1$. Its deviations are $-1,0,0,1$, so $\sigma_G^2=(1+0+0+1)/4=0.5$. Its third central moment is 0, and its fourth central moment is 0.5. Therefore $\gamma_{1,G}=0$ and $\gamma_{2,G}=0.5/(0.5)^2-3=-1$.

The error vector is $$ E_\mu=\frac{|1-1|}{1}=0,\quad E_{\sigma^2}=\frac{|0.5-1|}{1}=0.5,\quad E_{\gamma_1}=|0-0|=0,\quad E_{\gamma_2}=|-1-(-2)|=1. $$

The means and skewness match, but the generated variance is 50% below the target variance, and the excess kurtosis differs by 1. Matching a subset of summaries would have concealed these other discrepancies.

Kolmogorov–Smirnov Distance

Definition: For a distribution $A$, its cumulative distribution function is $F_A(x)=P(X\leq x)$. The two-sample Kolmogorov–Smirnov distance is the largest absolute vertical difference between the target and generated cumulative distribution functions: $$ D_{\mathrm{KS}}=\sup_x |F_G(x)-F_T(x)|. $$ It is dimensionless and lies between 0 and 1. For empirical distributions, calculate the difference over the step locations and take its largest absolute value.

The KS distance measures the greatest discrepancy in cumulative probability at any threshold. For example, if $D_{\mathrm{KS}}=0.25$, there is at least one threshold $x$ at which the fractions of generated and target values at or below $x$ differ by 0.25, and nowhere do they differ by more than 0.25. This is a descriptive distance, not a p-value or a hypothesis-test conclusion.

Worked Example: Empirical KS Distance

Let the hypothetical target values be $0,1,2,3$, and the generated values be $0,1,1,3$, each with equal weights. At $x=0$, both empirical CDFs equal $1/4$. At $x=1$, $F_T(1)=2/4$ while $F_G(1)=3/4$, so the absolute difference is $1/4$. At $x=2$, both CDFs equal $3/4$, and at $x=3$ both equal 1. Between these values, each CDF is constant, so no larger difference occurs.

Thus $D_{\mathrm{KS}}=\max_x |F_G(x)-F_T(x)|=1/4=0.25$. At the threshold of 1, the generated population has 25 percentage points more of its members at or below that value than the target population. The distance does not say how far apart the underlying values are in measurement units.

Wasserstein Distance

Definition: For one-dimensional distributions with finite first moments, the first Wasserstein distance is $$ W_1(G,T)=\int_{-\infty}^{\infty}|F_G(x)-F_T(x)|\,dx. $$ It has the same units as the variable. It can also be interpreted as the minimum average absolute distance over ways of pairing values from the two distributions, allowing probability mass to be transported between values.

For two equally sized empirical populations with $n$ values each, sort each population from smallest to largest. Pair values of the same rank. Then $$ W_1(G,T)=\frac{1}{n}\sum_{i=1}^{n}|x_{G,(i)}-x_{T,(i)}|, $$ where $x_{G,(i)}$ and $x_{T,(i)}$ are the $i$th sorted values. In one dimension, this ordered pairing achieves the minimum average absolute distance. The CDF-area formula gives the same result: each paired gap contributes its length to the area between the empirical CDFs, with overlapping gaps counted according to how many pairs cross each location.

Worked Example: Wasserstein Distance and Units

Consider hypothetical target values $0,2,4$ and generated values $1,2,6$, measured in milligrams. Both sets are already sorted. Pairing by rank gives absolute differences $|0-1|=1$, $|2-2|=0$, and $|4-6|=2$ mg.

Therefore, $$ W_1(G,T)=\frac{1+0+2}{3}=1\text{ mg}. $$

The interpretation is that the minimum average absolute movement needed to transform one empirical distribution into the other is 1 mg per population member. For comparison, the KS distance here is $1/3$: the CDF difference is $1/3$ from 0 to 1 mg and from 4 to 6 mg, and zero elsewhere. That dimensionless value does not distinguish a 1 mg gap from a much larger gap; Wasserstein distance reflects the size of the displacement.

Choosing and Diagnosing a Measure

These measures answer different questions, so a good comparison usually reports the measure that corresponds to the decision at hand and may report more than one. Standardized moment errors are compact and interpretable for selected features such as location, spread, and shape. They can be useful when those summaries have direct scientific meaning, but they depend on which moments are included. Two distributions can share mean and variance while differing substantially in shape.

The KS distance uses the entire univariate empirical CDF and identifies the largest cumulative-probability discrepancy. It is unchanged by a strictly increasing transformation of the measurement scale, but it does not reflect the physical distance between values. Wasserstein distance accounts for how far probability mass must move and is therefore sensitive to the scale and units: converting milligrams to micrograms multiplies $W_1$ by 1,000. The KS distance remains unchanged under that conversion.

Measure	What it emphasizes	Scale and units	Main caution
Standardized moment errors	Selected summaries such as mean, variance, skewness, and kurtosis	Dimensionless under the stated convention	Unreported features can still differ
KS distance	Largest CDF discrepancy at a threshold	Dimensionless	Does not encode the size of value gaps
Wasserstein distance $W_1$	Average displacement of distribution mass	Same units as the variable	Depends on scale and can be influenced by distant values

Diagnostics begin with checking that both distributions refer to the same variable and use the same units, weighting convention, and conditioning. Inspect the empirical distributions or CDFs as well as the final number: this helps locate where a discrepancy occurs and whether one or a few values dominate it. If target moments are estimated from data, their uncertainty also matters; a distance calculated against an estimated target is not a measure of certainty about the true population distribution.

A univariate distance cannot, on its own, establish that a multivariate virtual population has the right dependence among patient traits. Comparing each trait separately can miss differences in joint structure. Nor is there a universal cutoff that makes any one of these distances “small enough.” The acceptable discrepancy depends on the variable, its scientific role, the target’s precision, and how the virtual population will be used. These descriptive measures should not be treated as formal tests or as substitutes for scientific qualification.

Common Pitfalls

Reporting a standardized error without its denominator. A mean difference divided by the target standard deviation is not the same as one divided by the generated standard deviation. State the convention and use it consistently.
Calling KS distance a percentage of values that are wrong. It is the maximum difference between cumulative proportions at a threshold, not the fraction of individual records that disagree.
Interpreting Wasserstein distance as dimensionless. Its units are the units of the variable; state them and account for any scale conversion.
Assuming a small distance proves an adequate virtual population. The measure may omit important joint structure, target uncertainty, or scientifically consequential features. Interpret it alongside the purpose of the population and relevant diagnostics.

Key takeaway: State exactly how a distance is calculated, which features it captures, and its units. Moment errors summarize selected features, KS distance captures the largest CDF gap, and $W_1$ quantifies average one-dimensional displacement. None alone guarantees overall population adequacy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If a generated mean differs from the target mean by 0.4 units and the target standard deviation is 2 units, what is the standardized mean error under the convention used here?
At some threshold $x$, the target CDF is 0.60 and the generated CDF is 0.45. What does this tell you about their KS distance?
Two equal-size empirical distributions have sorted values $1,3$ and $2,5$, respectively. Calculate $W_1$ and state its units if the values are measured in hours.
Why can a KS distance of 0.1 not be interpreted as an average discrepancy of 0.1 measurement units?

Solutions. (1) $E_\mu=0.4/2=0.20$, a dimensionless error: the mean difference is one-fifth of the target standard deviation. (2) The absolute CDF difference at that threshold is $|0.60-0.45|=0.15$, so $D_{\mathrm{KS}}\geq 0.15$; the overall distance requires checking all thresholds. (3) The ordered absolute differences are $|1-2|=1$ and $|3-5|=2$ hours, so $W_1=(1+2)/2=1.5$ hours. (4) KS is a maximum difference in cumulative probabilities, so it is dimensionless and does not encode the size of gaps between values.