The Mean, Variance, and Covariance Kernel of the Empirical CDF

Why Study the Empirical CDF’s Covariance?

The empirical CDF is random because it is calculated from a random sample. Its value at one threshold fluctuates around the population probability below that threshold; values at different thresholds fluctuate together. The exact covariance between two such values describes this dependence, and can be derived directly from the indicator representation of the empirical CDF.

The formulas hold for any distribution, including distributions with atoms. The only sampling assumption needed is that $X_1,\ldots,X_n$ are independent and identically distributed with CDF $F$. We use the convention $F(x)=P(X_i\le x)$, so the threshold event includes observations equal to $x$.

Indicator Representation

Definition: For an independent, identically distributed sample $X_1,\ldots,X_n$ from a distribution with CDF $F$, the empirical CDF is $$ F_n(x)=\frac{1}{n}\sum_{i=1}^{n} I(X_i\le x),\qquad x\in\mathbb{R}, $$ where $I(A)=1$ if event $A$ occurs and $I(A)=0$ otherwise. Thus $F_n(x)$ is the fraction of sample observations at or below $x$.

For a fixed $x$, each $I(X_i\le x)$ is a Bernoulli random variable with success probability $F(x)$. Writing $F_n(x)$ as an average makes its first two moments accessible through indicator expectations. For two thresholds, the key fact is that their indicator events are nested.

Expectation and Variance

Theorem: For every $x\in\mathbb{R}$, $$ E[F_n(x)]=F(x),\qquad \operatorname{Var}(F_n(x))=\frac{F(x)\{1-F(x)\}}{n}. $$

Proof. Let $I_i(x)=I(X_i\le x)$. Since $E[I_i(x)]=P(X_i\le x)=F(x)$, linearity of expectation gives

E[F_n(x)] =\frac{1}{n}\sum_{i=1}^{n}E[I_i(x)] =\frac{1}{n}\,nF(x) =F(x).

An indicator with success probability $F(x)$ has variance $F(x)\{1-F(x)\}$, because $I_i(x)^2=I_i(x)$ and hence $E[I_i(x)^2]=F(x)$. The indicators for distinct sample observations are independent. Therefore their covariances vanish, and

\operatorname{Var}(F_n(x)) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(I_i(x)) =\frac{nF(x)\{1-F(x)\}}{n^2} =\frac{F(x)\{1-F(x)\}}{n}.

This proves the stated expectation and variance. In particular, the empirical CDF is unbiased for $F(x)$ at every fixed threshold. Its variance is zero if $F(x)$ is zero or one, and otherwise decreases as the sample size increases. $\square$

Worked Example: Two Thresholds in a Uniform Distribution

Suppose $X_1,\ldots,X_{10}$ are independent and uniformly distributed on $[0,1]$. Calculate the mean and variance of $F_{10}(0.3)$, and the covariance between $F_{10}(0.3)$ and $F_{10}(0.7)$.

For $0\le t\le 1$, the uniform CDF is $F(t)=t$. Thus $F(0.3)=0.3$, and

E[F_{10}(0.3)]=0.3,\qquad \operatorname{Var}(F_{10}(0.3)) =\frac{0.3(1-0.3)}{10} =\frac{0.21}{10} =0.021.

Because $0.3\le 0.7$, the covariance formula for ordered thresholds gives

\operatorname{Cov}(F_{10}(0.3),F_{10}(0.7)) =\frac{F(0.3)\{1-F(0.7)\}}{10} =\frac{0.3(1-0.7)}{10} =0.009.

The covariance is positive: observations counted at or below $0.3$ are also counted at or below $0.7$. The two empirical CDF values therefore share sample contributions, rather than varying independently.

The Covariance Kernel

Theorem: For any $x,y\in\mathbb{R}$, $$ \operatorname{Cov}(F_n(x),F_n(y)) =\frac{F(\min\{x,y\})-F(x)F(y)}{n}. $$ Equivalently, if $x\le y$, then $$ \operatorname{Cov}(F_n(x),F_n(y)) =\frac{F(x)\{1-F(y)\}}{n}. $$

Proof. Write $I_i(x)=I(X_i\le x)$ and $I_i(y)=I(X_i\le y)$. Expanding the covariance of the two averages gives

\operatorname{Cov}(F_n(x),F_n(y)) =\frac{1}{n^2}\sum_{i=1}^{n}\sum_{j=1}^{n} \operatorname{Cov}(I_i(x),I_j(y)).

For $i\ne j$, the indicators are functions of independent observations, so their covariance is zero. For $i=j$, the product of the indicators is one precisely when both threshold events occur. Their intersection is $\{X_i\le \min\{x,y\}\}$, and therefore

E[I_i(x)I_i(y)]=F(\min\{x,y\}).

Using $E[I_i(x)]=F(x)$ and $E[I_i(y)]=F(y)$, each same-observation covariance is

\operatorname{Cov}(I_i(x),I_i(y)) =F(\min\{x,y\})-F(x)F(y).

There are $n$ such terms in the double sum. Dividing their sum by $n^2$ proves the general formula. If $x\le y$, then $\min\{x,y\}=x$, so the formula becomes

\frac{F(x)-F(x)F(y)}{n} =\frac{F(x)\{1-F(y)\}}{n}.

The expression is symmetric in $x$ and $y$, as covariance must be; when $y\le x$, the corresponding ordered form is $F(y)\{1-F(x)\}/n$. $\square$

For $x\le y$, the single-observation events $A=\{X_i\le x\}$ and $B=\{X_i\le y\}$ satisfy $A\subseteq B$. Their indicator covariance is

\operatorname{Cov}(I(A),I(B)) =P(A)-P(A)P(B) =P(A)\{1-P(B)\}.

Thus the ordered-threshold covariance is positive or zero. In the formula, $F(x)\{1-F(y)\}$ is the product of the probability of being at or below $x$ and the probability of being above $y$. These are marginal probabilities in the covariance calculation, not the probability of one observation simultaneously belonging to both events. In fact, the events are disjoint when $x\le y$.

Worked Example: A Discrete Distribution With an Atom

Let $P(X=0)=0.2$, $P(X=2)=0.5$, and $P(X=5)=0.3$, and take an independent sample of size $n=4$. Find the means and variances at thresholds $1$ and $3$, and their covariance.

At threshold $1$, only the outcome $X=0$ is counted, so $F(1)=0.2$. At threshold $3$, the outcomes $0$ and $2$ are counted, so $F(3)=0.2+0.5=0.7$. The expectations are

E[F_4(1)]=0.2,\qquad E[F_4(3)]=0.7.

The two variances are

\operatorname{Var}(F_4(1)) =\frac{0.2(0.8)}{4} =0.04,\qquad \operatorname{Var}(F_4(3)) =\frac{0.7(0.3)}{4} =0.0525.

Since $1\le 3$, their covariance is

\operatorname{Cov}(F_4(1),F_4(3)) =\frac{0.2(1-0.7)}{4} =\frac{0.06}{4} =0.015.

The atom at $2$ increases the CDF from $0.2$ to $0.7$ between these thresholds. The formulas remain valid without any continuity assumption: they use the actual probabilities $P(X\le x)$, including any mass at or below the threshold.

Reading the Kernel and Checking Its Structure

The function

K_n(x,y)=\operatorname{Cov}(F_n(x),F_n(y)) =\frac{F(\min\{x,y\})-F(x)F(y)}{n}

is the finite-sample covariance kernel of the empirical CDF. Setting $y=x$ recovers the pointwise variance. Its dependence on $n$ is exactly $1/n$, while its dependence on the distribution and thresholds is contained in $F$. In particular, the kernel is determined by the population CDF, not by whether the distribution is continuous.

For any finite collection of thresholds $x_1,\ldots,x_m$, the matrix with entries $K_n(x_j,x_k)$ must be positive semidefinite. This is not an additional assumption: it follows because the matrix is a covariance matrix. To verify the claim directly, choose any real numbers $a_1,\ldots,a_m$. Then

\sum_{j=1}^{m}\sum_{k=1}^{m}a_ja_kK_n(x_j,x_k) =\operatorname{Var}\!\left(\sum_{j=1}^{m}a_jF_n(x_j)\right) \ge 0.

The equality follows by bilinearity of covariance, and the inequality follows from nonnegativity of variance. Thus the kernel’s entries cannot be chosen independently: all its finite covariance matrices obey this constraint.

Worked Example: Equal CDF Values Across a Gap

Suppose $P(X=0)=1/3$ and $P(X=1)=2/3$, with a sample of size $n=6$. Compare the empirical CDF values at $x=0$ and $y=1/2$.

The distribution has no mass between $0$ and $1$, so both thresholds count exactly the observations equal to $0$. In particular,

F(0)=F(1/2)=\frac{1}{3}.

The means are both $1/3$, and the variance at either threshold is

\frac{(1/3)(2/3)}{6} =\frac{2}{54} =\frac{1}{27}.

The covariance is also

\operatorname{Cov}(F_6(0),F_6(1/2)) =\frac{F(0)\{1-F(1/2)\}}{6} =\frac{(1/3)(2/3)}{6} =\frac{1}{27}.

Since the two empirical CDF values are in fact identical for every sample from this distribution, their covariance equals their common variance, as the calculation confirms.

Common Mistake

Confusing a covariance product with a joint-event probability. For $x\le y$, it is incorrect to describe $F(x)\{1-F(y)\}$ as the probability that one observation is both at or below $x$ and above $y$. Those events cannot occur together. The product appears after subtracting the product of the marginal means from the joint indicator expectation: $F(x)-F(x)F(y)=F(x)\{1-F(y)\}$.
Forgetting the factor $1/n$. Each observation contributes an indicator covariance, but averaging two sums introduces $1/n^2$; the $n$ same-observation terms leave an overall factor $1/n$.
Assuming continuity is required. The derivation uses only probabilities of threshold events and independence across observations. It is valid for discrete, continuous, and mixed distributions.

Key takeaway: For an independent, identically distributed sample, $F_n(x)$ is unbiased for $F(x)$, has variance $F(x)\{1-F(x)\}/n$, and has covariance with $F_n(y)$ equal to $\{F(\min\{x,y\})-F(x)F(y)\}/n$. The ordered-threshold form follows from nested events.

Check Your Understanding

Use the indicator representation and the covariance formula to answer the following.

For a sample of size $n$, what are $E[F_n(x)]$ and $\operatorname{Var}(F_n(x))$?
If $x\le y$, derive the ordered-threshold covariance from the general kernel formula.
Why is the joint probability of $\{X\le x\}$ and $\{X>y\}$ zero when $x\le y$, even though $F(x)\{1-F(y)\}$ appears in the covariance formula?
For $n=8$, $F(x)=0.25$, and $F(y)=0.75$ with $x\le y$, calculate $\operatorname{Cov}(F_n(x),F_n(y))$.
Why do the kernel formulas remain valid when the population distribution has an atom at a threshold?