Empirical CDFs for Distributional Comparison

Why Compare Cumulative Distributions?

A density plot emphasizes the shape of a distribution through a smoothed curve. An empirical cumulative distribution function (ECDF) instead answers a direct threshold question: what fraction of these values are at or below a specified value? This makes it useful for identifying where the observed and imputed distributions begin to differ, whether the difference persists across much of the scale, and whether the curves cross.

For a continuous variable in MICE, compare the values observed in the incomplete data with the values imputed for that variable. In a particular completed data set, the observed values are the same, while the imputed values may vary across imputations. The comparison is a diagnostic of the imputation model and the distributions it produces; it is not a direct comparison of two randomly assigned groups.

Definition: For values $y_1,\ldots,y_n$, the empirical cumulative distribution function is $$ F_n(x)=\frac{1}{n}\sum_{i=1}^{n} I(y_i\le x), $$ where $I(y_i\le x)$ equals 1 when $y_i\le x$, and 0 otherwise. Thus $F_n(x)$ is the proportion of the sample at or below $x$. It is a nondecreasing, right-continuous step function that approaches 0 below the smallest value and equals 1 at or above the largest value.

Let $F_{\mathrm{obs}}(x)$ denote the ECDF of observed values and $F_{\mathrm{imp}}(x)$ the ECDF of imputed values in one completed data set. Define their signed vertical difference as

D(x)=F_{\mathrm{imp}}(x)-F_{\mathrm{obs}}(x).

When $D(x)>0$, a larger fraction of imputed values than observed values is at or below $x$. At that threshold, the imputed values therefore have relatively more mass on the lower side. When $D(x)<0$, a smaller fraction of imputed values is at or below $x$, so they have relatively more mass above the threshold. The sign describes a cumulative difference at $x$; it does not say that every individual imputed value is lower or higher.

Reading the Curves and Their Gaps

The two ECDFs should be drawn on the same horizontal scale and with the vertical axis running from 0 to 1. Each jump represents one or more values at that location. If the two curves track closely, their empirical cumulative proportions are similar over the range. If they separate, the gap identifies thresholds at which the proportions differ. A persistent positive gap suggests relatively smaller imputed values over that range; a negative gap suggests relatively larger imputed values. A gap that changes sign indicates that the distributions do not differ in a single direction across all thresholds.

A useful descriptive summary is the largest absolute vertical gap,

D_{\max}=\sup_x |F_{\mathrm{imp}}(x)-F_{\mathrm{obs}}(x)|.

For finite samples, the supremum is attained at one or more observed or imputed data values. It reports the largest difference in cumulative proportions, but not where the difference occurs; inspect the curves as well as any summary. Neither a small gap nor a large one has a universal pass/fail threshold. Its practical importance depends on the variable, the sample sizes, and how the variable is used in the analysis.

Worked Example: Calculating an ECDF Gap

Suppose a hypothetical diagnostic has five observed values, in measurement units: 2, 4, 5, 8, 10. One imputed data set has five imputed values: 1, 3, 5, 6, 9. Calculate the ECDFs at selected thresholds and interpret the differences.

At $x=3$, two imputed values (1 and 3) are at or below 3, while one observed value (2) is. Therefore

F_{\mathrm{imp}}(3)=\frac{2}{5}=0.400,\qquad F_{\mathrm{obs}}(3)=\frac{1}{5}=0.200,\qquad D(3)=0.400-0.200=0.200.

At $x=6$, four imputed values (1, 3, 5, 6) and three observed values (2, 4, 5) are at or below 6:

F_{\mathrm{imp}}(6)=\frac{4}{5}=0.800,\qquad F_{\mathrm{obs}}(6)=\frac{3}{5}=0.600,\qquad D(6)=0.800-0.600=0.200.

The imputed distribution has a 0.200 larger cumulative proportion at each of these thresholds. Checking the jump locations gives a largest absolute gap of 0.200 for these data. This is a descriptive difference of 20 percentage points in cumulative proportions, not a claim that each imputed value is 20% lower.

Why the ECDF Has These Properties

The step-function form follows directly from the indicator representation: as $x$ increases, each indicator can change only from 0 to 1, at its corresponding observed value. Consequently the sum cannot decrease, and the function changes only at sample values. With no tied values, each jump is $1/n$; a tie produces a jump equal to the number tied divided by $n$.

There is also a simple population justification. If $Y_1,\ldots,Y_n$ are independent draws from a common distribution with CDF $F(x)=P(Y\le x)$, then for a fixed $x$, each indicator $I(Y_i\le x)$ is a Bernoulli variable with success probability $F(x)$. Their average is the ECDF, so

E\{F_n(x)\}=F(x),\qquad \operatorname{Var}\{F_n(x)\}=\frac{F(x)\{1-F(x)\}}{n}.

This explains why ECDFs estimate cumulative probabilities and why a finite sample gives a stepwise approximation. The calculation assumes independent, identically distributed draws. Observed and imputed values in a MICE diagnostic do not automatically satisfy that sampling setup: the observed values may be selected by the missingness process, and imputations are generated using fitted models and may be dependent across iterations or chains. The formulas justify the ECDF as a sample summary, not a ready-made standard error or significance test for the observed-imputed gap.

Worked Example: Locating a Broad Difference

Consider hypothetical observed values $10,12,13,14,15,16,18,20$ and imputed values $10,11,12,13,13,14,15,16$, all in the same units. Each group has eight values. Compare cumulative proportions at four thresholds.

At $x=12$, two observed values and three imputed values are at or below the threshold. Thus $F_{\mathrm{obs}}(12)=2/8=0.250$, $F_{\mathrm{imp}}(12)=3/8=0.375$, and $D(12)=0.375-0.250=0.125$.

At $x=14$, four observed values and six imputed values are at or below the threshold. Hence $F_{\mathrm{obs}}(14)=4/8=0.500$, $F_{\mathrm{imp}}(14)=6/8=0.750$, and $D(14)=0.250$.

At $x=16$, the corresponding counts are six and eight, giving $F_{\mathrm{obs}}(16)=6/8=0.750$, $F_{\mathrm{imp}}(16)=8/8=1.000$, and $D(16)=0.250$. At $x=18$, the counts are seven and eight, so $D(18)=1.000-0.875=0.125$.

The positive gaps at all four thresholds indicate relatively more imputed values at or below these values. Here the imputed values are concentrated toward the lower part of the observed range. This pattern is consistent with a broad lower shift in this small example, but the ECDF alone does not establish why it occurred or whether the imputation model is wrong.

Conditions for a Meaningful Diagnostic

An overlay is interpretable only if both samples represent the intended comparison. Check that the values have the same units, coding, and transformations; that the observed group contains observed values of the variable and the imputed group contains imputations for the same variable; and that the plot labels make clear which is which. Use a shared horizontal and vertical scale. Different sample sizes are allowed, but they affect the granularity of the steps: a group of size $n$ has ECDF increments of $1/n$ when its values are distinct.

The key assumption for treating similar marginal distributions as an expectation is that the two groups are comparable with respect to the target distribution. Under missing completely at random (MCAR), observed values can represent the marginal distribution of the variable among all cases, subject to sampling variation. Under missing at random (MAR), the probability of being observed may depend on recorded variables. The observed and missing cases can then have different marginal distributions even when the conditional imputation model is appropriate. An overall ECDF discrepancy may reflect that difference in composition, not necessarily a defect in the conditional model.

For each completed data set, compare the observed values with that data set's imputed values. Repeat the comparison across imputations to see whether a discrepancy is consistent or varies substantially. Do not concatenate all $m$ sets of imputed values and treat the resulting $m$-fold collection as $m$ times as many independent missing cases. The completed data sets represent uncertainty about missing values; they are not independent replications that multiply the amount of observed information.

Define the two samples.
Identify observed and imputed values for the same continuous variable, in common units and on a common scale.

Draw comparable ECDFs.
Use a shared horizontal scale and vertical range from 0 to 1; label the imputation or completed data set.

Read the signed gaps.
At thresholds where curves differ, calculate or estimate $D(x)$; note its sign, size, and whether it changes sign.

Check consistency and context.
Look across imputations and consider missingness and predictor composition before interpreting an overall difference as model failure.

Worked Example: Curve Crossings Across Imputations

Suppose the hypothetical observed values are $2,4,6,8$. Three separate imputations produce $1,3,5,7$; $3,5,7,9$; and $2,4,6,10$. Compare each set with the observed ECDF at $x=4$ and $x=6$, rather than pooling the three imputed sets.

At $x=4$, the observed count is $2$, so $F_{\mathrm{obs}}(4)=2/4=0.500$. The first imputation has two values at or below 4, giving $F_{\mathrm{imp},1}(4)=0.500$ and $D_1(4)=0$. The second has one, giving $F_{\mathrm{imp},2}(4)=0.250$ and $D_2(4)=-0.250$. The third has two, giving $F_{\mathrm{imp},3}(4)=0.500$ and $D_3(4)=0$.

At $x=6$, the observed count is $3$, so $F_{\mathrm{obs}}(6)=0.750$. The first imputation has three values at or below 6, giving $D_1(6)=0.750-0.750=0$. The second has two, giving $D_2(6)=0.500-0.750=-0.250$. The third has three, giving $D_3(6)=0$.

The second imputation has fewer values below both thresholds, while the first and third match the observed cumulative proportions at these points. This example illustrates why one plot should not stand in for all imputations. It also shows that a difference at selected thresholds is not a complete description of a curve: inspect the entire range before describing its pattern.

Diagnostics, Alternatives, and Common Pitfalls

A useful ECDF diagnostic is readable and reproducible: the axes are labelled, the same scale is used across plots, the observed and imputed groups are clearly distinguished, and the particular completed data set or chain is identified. Look for systematic separation, an abrupt gap near a particular range, frequent crossings, or substantial variation in the gap across imputations. These are prompts to investigate the model, coding, missingness pattern, and relevant predictors—not automatic evidence of a specific failure.

ECDFs complement other distribution displays. Density plots can make broad shape differences visually apparent but depend on smoothing choices. Quantile plots align selected distributional landmarks, whereas an ECDF displays cumulative proportions at every threshold and can reveal where a difference arises between those landmarks. A numerical maximum-gap summary compresses the plot into one number, but loses location and direction. These views answer related but different questions; none alone validates an imputation model.

A common mistake is to see a positive $D(x)$ and conclude that “the imputations are too low.” The correct statement is narrower: at threshold $x$, a larger proportion of the imputed sample is at or below $x$ than of the observed sample. Before attributing this to model failure, consider whether observed and missing cases differ in recorded predictors and whether the marginal comparison is appropriate. Another mistake is to apply a two-sample Kolmogorov–Smirnov p-value as though the groups were independent random samples from a common distribution. The standard test's reference distribution does not automatically account for MICE's data-dependent imputations, missingness selection, or repeated imputed values across completed data sets. A plotted discrepancy is best treated here as a descriptive diagnostic, not a calibrated test.

ECDF overlays have limitations. They do not show which predictors explain a discrepancy, distinguish a misspecified conditional model from different case composition, or prove that the unobserved values follow the imputed distribution. Sparse observed data also produce coarse steps, so apparent gaps can be unstable. Conversely, close marginal curves do not guarantee that the conditional relationships used in later analyses are well represented. Interpret the plot alongside the imputation context and other diagnostics.

Key takeaway: An ECDF overlay shows how observed and imputed cumulative proportions compare across thresholds. The signed gap gives direction at a threshold; the pattern across thresholds and across imputations provides the diagnostic context. Similarity is reassuring only to the extent that the groups are meaningfully comparable.

Check Your Understanding

Answer each question, then compare with the solutions below.

For observed values $1,3,5$ and imputed values $2,3,6$, calculate $D(3)$.
What does $D(x)=-0.20$ mean at a particular threshold $x$?
Why should imputed values from several completed data sets not simply be pooled and treated as independent observations?
Under MAR, can observed and imputed values have different marginal ECDFs even if the conditional imputation model is appropriate? Explain.

Solutions. (1) Two values in each group are at or below 3, so $F_{\mathrm{imp}}(3)=2/3$, $F_{\mathrm{obs}}(3)=2/3$, and $D(3)=0$. (2) The imputed sample has a cumulative proportion 0.20 lower than the observed sample at or below $x$; equivalently, relatively more imputed values are above that threshold. (3) The imputations represent uncertainty across completed data sets, not additional independent missing cases; pooling them this way overstates the amount of information and ignores their dependence and shared observed data. (4) Yes. Under MAR, observation can depend on recorded variables, so observed and missing cases may differ in their predictor composition and hence in their marginal distributions, even when their conditional distributions are modeled appropriately.