Empirical CDF Values at Observations and Their Ranks

Why Evaluate the Empirical CDF at the Data?

The empirical CDF records the fraction of sample observations at or below a given point. Evaluating it at a sample observation makes that fraction especially concrete: it counts how many observations are no larger than that one. When the sample has no ties, this count is exactly the observation’s rank. The identity links an empirical-distribution value to the ordering of the data, and it gives an exact description of the values produced across the sample.

Throughout, let $X_1,\ldots,X_n$ be independent and identically distributed real-valued random variables with a continuous distribution function $F$. Write their empirical CDF as $F_n$. Continuity ensures that two independent observations are equal with probability zero, so the sample has no ties with probability one. No density or strict increase of $F$ is required.

Definition: The empirical CDF of $X_1,\ldots,X_n$ is $$ F_n(x)=\frac{1}{n}\sum_{j=1}^n \mathbf{1}\{X_j\leq x\}, \qquad x\in\mathbb{R}. $$ When the observations are distinct, the ascending rank of $X_i$ is $$ R_i=\sum_{j=1}^n\mathbf{1}\{X_j\leq X_i\}. $$ Thus rank 1 is the smallest observation and rank $n$ is the largest.

The Rank Identity

Substituting a sample observation $X_i$ into the definition of $F_n$ counts the observations no larger than $X_i$. With no ties, that count is precisely its ascending rank.

Theorem: If $X_1,\ldots,X_n$ are iid from a continuous distribution, then, with probability one, for every $i=1,\ldots,n$, $$ F_n(X_i)=\frac{R_i}{n}. $$ Moreover, the rank vector $(R_1,\ldots,R_n)$ is a uniformly distributed random permutation of $(1,\ldots,n)$.

Proof. For any fixed $i$, the definition gives $$ F_n(X_i)=\frac{1}{n}\sum_{j=1}^n\mathbf{1}\{X_j\leq X_i\}=\frac{R_i}{n}. $$ Because the common distribution is continuous, $P(X_j=X_k)=0$ for each pair $j\ne k$. There are finitely many pairs, so the probability of any tie in the sample is zero.

To establish the rank-vector law, consider the $n!$ possible strict orderings of the labelled observations. They are disjoint, and together they have probability one because ties have probability zero. The joint distribution of an iid sample is unchanged by permuting its labels. Consequently, each strict ordering has the same probability; since the probabilities sum to one, each is $1/n!$. Every strict ordering corresponds to exactly one permutation of the rank vector. Thus the rank vector is uniform over all permutations of $(1,\ldots,n)$. $\square$

Worked Example: A Specified Observation in a Sample of Four

Let $X_1,\ldots,X_4$ be iid $N(0,1)$, and consider $F_4(X_2)$. Find its distribution and calculate $P(F_4(X_2)\leq 1/2)$.

With probability one, $X_2$ has one of the ranks 1, 2, 3, or 4. By the theorem, its rank is uniform over these four possibilities, and the empirical CDF value is its rank divided by 4. Therefore, $$ P\left(F_4(X_2)=\frac{r}{4}\right)=\frac14,\qquad r=1,2,3,4. $$

The event $F_4(X_2)\leq 1/2$ occurs precisely when the rank is 1 or 2. Hence $$ P\left(F_4(X_2)\leq\frac12\right) =P(R_2\in\{1,2\}) =\frac{2}{4} =\frac12. $$ The answer uses continuity and identical sampling, not a special property of the normal distribution.

Marginal and Joint Laws

For any fixed label $i$, the uniform rank-vector result implies that $R_i$ is uniform on $\{1,\ldots,n\}$. The corresponding empirical CDF value therefore has a discrete uniform distribution on the grid $\{1/n,2/n,\ldots,1\}$. These marginal laws do not mean that the $n$ values are independent: the sample’s ranks must occupy every rank exactly once.

Corollary: For each fixed $i$ and $r\in\{1,\ldots,n\}$, $$ P\left(F_n(X_i)=\frac{r}{n}\right)=\frac1n. $$ For distinct labels $i$ and $j$, and $r,s\in\{1,\ldots,n\}$, $$ P\left(F_n(X_i)=\frac{r}{n},\,F_n(X_j)=\frac{s}{n}\right) = \begin{cases} \dfrac{1}{n(n-1)}, & r\ne s,\\[4pt] 0, & r=s. \end{cases} $$

To see the joint probabilities, there are $n!$ equally likely rank permutations. If $r\ne s$, fixing $R_i=r$ and $R_j=s$ leaves $(n-2)!$ ways to assign the other ranks. The probability is therefore $(n-2)!/n!=1/[n(n-1)]$. If $r=s$, two distinct observations cannot have the same rank in a tie-free sample, so the probability is zero.

Worked Example: Two Specified Values in a Sample of Five

Let $X_1,\ldots,X_5$ be iid from any continuous distribution. Find the probability that $F_5(X_1)=2/5$ and $F_5(X_4)=4/5$. Also find the probability that both values equal $2/5$.

The first event fixes two distinct labels at two distinct ranks. There are $5!=120$ equally likely rank permutations, and fixing these ranks leaves $3!=6$ assignments for the other observations. Thus $$ P\left(F_5(X_1)=\frac25,\,F_5(X_4)=\frac45\right) =\frac{3!}{5!} =\frac{6}{120} =\frac1{20}. $$

For the second event, both observations would have rank 2. That is impossible when there are no ties, so $$ P\left(F_5(X_1)=\frac25,\,F_5(X_4)=\frac25\right)=0. $$ By contrast, each one of these labels separately has probability $1/5$ of taking the value $2/5$. The marginal probabilities alone do not describe the joint behavior.

The Whole Collection and Its Dependence

Although the value at a fixed label is random, the collection of all values across the sample is not random when considered without labels. In every tie-free sample, exactly one observation has each rank from 1 through $n$. Equivalently, sorting the observations and evaluating $F_n$ at them gives a deterministic sequence.

Key identity: Write $X_{(1)}<\cdots<X_{(n)}$ for the order statistics, in ascending order. With probability one, $$ F_n(X_{(k)})=\frac{k}{n},\qquad k=1,\ldots,n. $$ For a fixed label $i$, $F_n(X_i)$ is uniform on the same grid, while the vector $(F_n(X_1),\ldots,F_n(X_n))$ is a uniformly random permutation of that grid.

The dependence between labels can also be summarized through moments. Since a fixed value is uniform on the grid, its mean and variance follow by summing the first $n$ integers and their squares: $$ E[F_n(X_i)]=\frac{1}{n}\sum_{r=1}^n\frac{r}{n}=\frac{n+1}{2n}, \qquad \operatorname{Var}(F_n(X_i))=\frac{n^2-1}{12n^2}. $$ For $i\ne j$, the sum of all $n$ empirical CDF values is always $(n+1)/2$. Thus the sum of their centered values is zero. Squaring that centered sum, taking expectations, and using that all distinct-label covariances are equal by symmetry gives $$ 0=n\operatorname{Var}(F_n(X_i))+n(n-1)\operatorname{Cov}(F_n(X_i),F_n(X_j)). $$ Therefore, $$ \operatorname{Cov}(F_n(X_i),F_n(X_j)) =-\frac{n^2-1}{12n^2(n-1)} =-\frac{n+1}{12n^2}. $$ The negative covariance expresses the rank constraint: if one label takes a high rank, fewer high ranks remain available to the others.

Worked Example: The Collection and Covariance for Three Observations

Suppose $X_1,X_2,X_3$ are iid continuous. The three values $F_3(X_i)$, with labels ignored, must be $1/3,2/3,1$, one each. For a specified label, such as $i=1$, each of these values has probability $1/3$.

For two distinct labels, the joint value $F_3(X_1)=1/3$ and $F_3(X_2)=2/3$ has probability $$ \frac{1}{3(3-1)}=\frac16. $$ The same probability applies to any ordered pair of distinct grid values; equal values have probability zero.

The marginal mean and variance are $$ E[F_3(X_1)]=\frac{1+2+3}{9}=\frac23, \qquad \operatorname{Var}(F_3(X_1)) =\frac{3^2-1}{12\cdot 3^2} =\frac{2}{27}. $$ For distinct labels, the covariance is $$ \operatorname{Cov}(F_3(X_1),F_3(X_2)) =-\frac{3+1}{12\cdot 3^2} =-\frac{1}{27}. $$ The covariance is negative even though the original observations are independent: the dependence arises from evaluating the shared empirical CDF at the sample points.

Why Continuity Matters

The identity $F_n(X_i)=R_i/n$ uses the rank convention for a tie-free sample. If ties occur and rank means the usual position in an ordered list with tied observations occupying consecutive positions, the equality need not hold for every tied observation. For example, in the realized sample $2,2,5$, the empirical CDF at either observation equal to 2 is $2/3$, while their occupied ranks are 1 and 2. The no-ties condition is therefore substantive, not cosmetic.

A common mistake is to treat the values $F_n(X_1),\ldots,F_n(X_n)$ as independent because each has the same discrete uniform marginal distribution. That reasoning overlooks that their ranks form a permutation: no two values can coincide, and every grid value must appear. Use the marginal uniform law for a single specified label; use the joint rank law or permutation constraint for questions involving multiple labels.

Key takeaway: For an iid continuous sample, the empirical CDF at a sampled point is its rank divided by $n$. At a fixed label the value is discrete uniform on the rank grid, but across labels the values are a random permutation of that grid, not independent draws.

Check Your Understanding

Use the rank identity and the random-permutation law to answer these questions.

For an iid continuous sample of size 8, what is the distribution of $F_8(X_3)$?
For an iid continuous sample of size 6, find $P(F_6(X_1)=2/6,\ F_6(X_5)=5/6)$.
In a tie-free sample of size $n$, what is $F_n(X_{(k)})$ for a specified $k$?
Why is it impossible for two distinct labels to have equal empirical CDF values in a tie-free sample?
For $n=4$, find the covariance of $F_4(X_1)$ and $F_4(X_2)$, assuming iid continuous observations.