Pointwise Convergence of the Empirical Distribution Function

Why a Fixed Threshold Matters

A sample can be used to estimate not only a mean or a proportion, but the entire distribution of a random variable. At a chosen threshold $x$, the relevant question is simple: what fraction of observations are at most $x$? That fraction is the empirical distribution function at $x$. Its convergence to the population probability at that threshold is a direct application of the law of large numbers.

The key is to hold $x$ fixed. Then each observation contributes either 1 or 0 according to whether it is at most $x$. The empirical CDF is therefore an average of IID indicator variables, and its limit follows from the strong law. This pointwise result does not, by itself, establish convergence simultaneously over every real threshold.

Definition and Indicator Representation

Definition: Let $X_1,X_2,\ldots$ be IID real-valued random variables with common distribution function $F(x)=P(X_1\leq x)$. For $n\geq1$, the empirical distribution function is $$ F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\{X_i\leq x\},\qquad x\in\mathbb{R}, $$ where $\mathbf{1}\{A\}$ equals 1 when event $A$ occurs and 0 otherwise. Thus $F_n(x)$ is the proportion of the first $n$ observations that are at most $x$.

For any fixed real number $x$, define $Y_i(x)=\mathbf{1}\{X_i\leq x\}$. The event $\{X_i\leq x\}$ is measurable because $X_i$ is a random variable and $(-\infty,x]$ is a Borel set. Independence and identical distribution of the $X_i$'s imply that $Y_1(x),Y_2(x),\ldots$ are IID. Each has a Bernoulli distribution with success probability $F(x)$, and $$ F_n(x)=\frac{1}{n}\sum_{i=1}^{n}Y_i(x). $$ This identity is the central step: at a fixed threshold, estimating a CDF value is estimating the mean of a bounded IID sequence.

Worked Example: A Uniform Distribution at a Fixed Threshold

Suppose $X_1,X_2,\ldots$ are IID Uniform$(0,1)$, and consider $x=0.4$. For one observation, $$ P(X_i\leq 0.4)=F(0.4)=0.4. $$

The indicator $Y_i(0.4)=\mathbf{1}\{X_i\leq0.4\}$ is therefore Bernoulli with mean $0.4$ and variance $0.4(1-0.4)=0.24$. The empirical CDF is the average of these indicators: $$ F_n(0.4)=\frac{1}{n}\sum_{i=1}^{n}Y_i(0.4). $$ Its expectation is $0.4$, and its variance is $0.24/n$. For example, with $n=1000$, Chebyshev's inequality gives $$ P\bigl(|F_{1000}(0.4)-0.4|\geq0.05\bigr) \leq \frac{0.24}{1000(0.05)^2} =\frac{0.24}{2.5}=0.096. $$ The empirical proportion is thus centered at the target $0.4$, and this bound shows how its deviations become less likely as the sample size grows.

Almost-Sure Convergence at a Fixed Threshold

The bounded IID strong law states that if $Y_1,Y_2,\ldots$ are IID and $E[|Y_1|]<\infty$, then $n^{-1}\sum_{i=1}^{n}Y_i\xrightarrow{a.s.}E[Y_1]$. Here each indicator is bounded by 1, so the integrability condition holds. Applying that law gives the pointwise empirical-CDF result.

Theorem: If $X_1,X_2,\ldots$ are IID with common CDF $F$, then for every fixed $x\in\mathbb{R}$, $$ F_n(x)\xrightarrow{a.s.}F(x). $$

Proof. Fix $x\in\mathbb{R}$, and set $Y_i=\mathbf{1}\{X_i\leq x\}$. These variables are IID, $0\leq Y_i\leq1$, and $$ E[Y_i]=1\cdot P(X_i\leq x)+0\cdot P(X_i>x)=F(x). $$ By the bounded IID strong law, $n^{-1}\sum_{i=1}^{n}Y_i\xrightarrow{a.s.}F(x)$. The definition of $F_n(x)$ makes this average exactly $F_n(x)$, proving the claim for the fixed threshold. $\square$

No density, continuity, or moment assumption on the distribution of $X_i$ is needed. In particular, the theorem applies whether the population distribution is continuous, discrete, or a mixture. The threshold is included in the event: the definition uses $X_i\leq x$, not $X_i<x$.

Worked Example: An Exponential Distribution

Let $X_i$ be IID exponential random variables with rate 2, so that $F(x)=1-e^{-2x}$ for $x\geq0$. Choose $x=(\log 2)/2$. Since $2x=\log 2$, $$ F(x)=1-e^{-\log 2}=1-\frac{1}{2}=\frac{1}{2}. $$

At this threshold, $Y_i=\mathbf{1}\{X_i\leq(\log2)/2\}$ is Bernoulli with success probability $1/2$, and $$ F_n\bigl((\log2)/2\bigr)=\frac{1}{n}\sum_{i=1}^{n}Y_i. $$ The strong law gives $F_n((\log2)/2)\xrightarrow{a.s.}1/2$. For a finite-sample calculation, the variance is $(1/2)(1/2)/n=1/(4n)$. At $n=100$, Chebyshev's inequality yields $$ P\left(\left|F_{100}\bigl((\log2)/2\bigr)-\frac{1}{2}\right|\geq0.1\right) \leq\frac{1/400}{(0.1)^2} =\frac{0.0025}{0.01}=0.25. $$ The exponential model determines the target probability, but the convergence argument itself uses only the indicator representation and the law of large numbers.

Expectation, Variance, and Convergence in Probability

The same representation gives an exact finite-sample description. For fixed $x$, each $Y_i(x)$ is Bernoulli with success probability $p=F(x)$. Its expectation is $p$ and its variance is $p(1-p)$. Independence makes the covariances between distinct indicators zero, so averaging gives the following result.

Result: For every fixed $x\in\mathbb{R}$ and every $n\geq1$, $$ E[F_n(x)]=F(x),\qquad \operatorname{Var}(F_n(x))=\frac{F(x)\{1-F(x)\}}{n}. $$ Consequently, for every $\varepsilon>0$, $$ P\bigl(|F_n(x)-F(x)|\geq\varepsilon\bigr) \leq\frac{F(x)\{1-F(x)\}}{n\varepsilon^2} \leq\frac{1}{4n\varepsilon^2}. $$

Derivation. Write $p=F(x)$ and $Y_i=\mathbf{1}\{X_i\leq x\}$. Linearity of expectation gives $$ E[F_n(x)]=\frac{1}{n}\sum_{i=1}^{n}E[Y_i]=\frac{1}{n}(np)=p. $$ Since the $Y_i$'s are independent and each has variance $p(1-p)$, $$ \operatorname{Var}(F_n(x)) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(Y_i) =\frac{np(1-p)}{n^2} =\frac{p(1-p)}{n}. $$ Chebyshev's inequality then gives the first probability bound. Finally, $p(1-p)\leq1/4$ for $0\leq p\leq1$, because $p(1-p)=1/4-(p-1/2)^2$. This proves the second bound as well. $\square$

For fixed $x$, the right-hand side tends to zero as $n$ increases, so $F_n(x)\xrightarrow{p}F(x)$. This weak-law conclusion also follows from the almost-sure result, but the variance calculation provides an explicit bound at each sample size. If $F(x)=0$ or $F(x)=1$, the indicator is constant almost surely and $F_n(x)=F(x)$ almost surely for every $n$.

Worked Example: A Discrete Distribution and the Included Endpoint

Suppose $X_i$ are IID with $$ P(X_i=-1)=0.2,\qquad P(X_i=2)=0.5,\qquad P(X_i=5)=0.3. $$ At $x=2$, the event in the empirical CDF is $X_i\leq2$. It includes both $-1$ and $2$, so $$ F(2)=P(X_i=-1)+P(X_i=2)=0.2+0.5=0.7. $$

Thus $F_n(2)$ is the fraction of observations equal to $-1$ or $2$. Its indicator representation has success probability $0.7$, giving $$ E[F_n(2)]=0.7,\qquad \operatorname{Var}(F_n(2))=\frac{0.7(0.3)}{n}=\frac{0.21}{n}. $$ For $n=100$, the probability that $F_{100}(2)$ differs from $0.7$ by at least $0.1$ is bounded by $$ P(|F_{100}(2)-0.7|\geq0.1) \leq\frac{0.21/100}{0.1^2} =\frac{0.0021}{0.01}=0.21. $$ The strong law also gives $F_n(2)\xrightarrow{a.s.}0.7$. At any threshold strictly between 2 and 5, the CDF value is still $0.7$; at a threshold strictly below 2 but at least $-1$, it is $0.2$. This jump illustrates why the threshold convention and the distribution's atoms matter.

What Pointwise Convergence Does and Does Not Say

The theorem's quantifiers are important: for each fixed $x$, the probability that $F_n(x)$ converges to $F(x)$ is 1. The null set that may be excluded in that assertion can depend on $x$. It is not valid to take an uncountable collection of probability-one events and conclude, merely from those separate statements, that their intersection has probability one. A simultaneous assertion over all thresholds requires an additional argument and is a different, stronger conclusion.

Pointwise convergence is nevertheless useful on its own. At any prespecified cutoff—such as a clinical threshold, a percentile target, or a boundary used to define an event—the empirical fraction below that cutoff consistently estimates the corresponding population probability. The result also explains why the empirical CDF works for discrete distributions: at a fixed threshold, an atom simply changes the Bernoulli success probability.

Common Mistakes

Using the wrong endpoint. The empirical CDF counts observations satisfying $X_i\leq x$. Replacing this by $X_i<x$ can change the answer when there is positive probability at $x$.
Calling the indicators identically distributed without fixing the threshold. The argument defines $Y_i(x)$ for one fixed $x$. For that $x$, the indicators are IID Bernoulli$(F(x))$.
Confusing pointwise with simultaneous convergence. Separate probability-one convergence statements for each fixed threshold do not alone prove convergence at every threshold on one common probability-one event.
Assuming a continuous distribution is required. The proof needs only IID observations and bounded indicators; atoms and mixed distributions are allowed.

Key takeaway: At each fixed $x$, the empirical CDF is the sample average of the IID indicators $\mathbf{1}\{X_i\leq x\}$. Their mean is $F(x)$, so the strong law gives almost-sure pointwise convergence, while their variance gives an explicit Chebyshev bound.

Check Your Understanding

Use the indicator representation to answer each question.

For a fixed $x$, what are the distribution, expectation, and variance of $\mathbf{1}\{X_i\leq x\}$?
If $F(x)=0.3$, calculate $E[F_n(x)]$ and $\operatorname{Var}(F_n(x))$.
Why does the pointwise strong-law argument apply when the population distribution has an atom at $x$?
For the discrete distribution with masses $0.2,0.5,0.3$ at $-1,2,5$, what event does $F_n(2)$ count?
Why does proving convergence almost surely for each fixed threshold not, by itself, prove convergence simultaneously for all real thresholds?