The Probability Integral Transform for Order Statistics

Why Transform an Order Statistic?

An order statistic depends on both the sample size and the parent distribution. For instance, the sample minimum from an exponential population and the sample minimum from a normal population have different distributions. Yet applying the parent cumulative distribution function (CDF) to an observation puts it on a common probability scale. For an iid sample from a continuous distribution, the transformed $k$th order statistic has a beta distribution that does not depend on the parent distribution.

This result turns calculations about the location of an order statistic into calculations about a uniform sample. It also lets us evaluate probabilities involving a population quantile without specifying the shape of the parent distribution.

Definition: For a CDF $F$, the probability integral transform of a random variable $X$ is $F(X)$. If $X_1,\ldots,X_n$ are arranged in increasing order, write $X_{(1)}\le\cdots\le X_{(n)}$ for their order statistics. The transformed $k$th order statistic is $F(X_{(k)})$, where $1\le k\le n$.

From a Continuous CDF to Uniform Variables

We first establish the needed probability integral transform. Continuity matters: for a continuous CDF, $F(X)$ is uniform on $(0,1)$, even if $F$ is not strictly increasing. A flat section of $F$ creates no probability mass, so it does not prevent the transform from being uniform.

Proposition: If $X$ has a continuous CDF $F$, then $U=F(X)\sim\operatorname{Uniform}(0,1)$.

Proof. Fix $u\in(0,1)$, and let $b=\sup\{x:F(x)\le u\}$. Since a CDF tends to 1 as $x$ tends to infinity, $b$ is finite. Continuity and monotonicity imply $F(b)=u$. They also imply that $F(x)\le u$ exactly when $x\le b$. Therefore

P(F(X)\le u)=P(X\le b)=F(b)=u.

The resulting CDF equals the uniform CDF for every $u\in(0,1)$; it is 0 for $u\le0$ and 1 for $u\ge1$. Hence $F(X)$ is uniform. $\square$

Consequently, if $X_1,\ldots,X_n$ are independent and identically distributed (iid) with continuous CDF $F$, then $U_i=F(X_i)$ are iid uniform variables. Because $F$ is nondecreasing, arranging the observations in increasing order also arranges their transformed values. Thus $F(X_{(k)})=U_{(k)}$ almost surely. The order-statistic calculation can now be made on the uniform scale.

The Beta Law

For positive parameters $a,b$, the beta distribution $\operatorname{Beta}(a,b)$ has density

f(u)=\frac{u^{a-1}(1-u)^{b-1}}{B(a,b)},\qquad 0<u<1,

where $B(a,b)=\int_0^1 t^{a-1}(1-t)^{b-1}\,dt$ is the beta function. For positive integers $a,b$, $B(a,b)=(a-1)!(b-1)!/(a+b-1)!$.

Theorem: Let $X_1,\ldots,X_n$ be iid with continuous CDF $F$. For $1\le k\le n$, $$ F(X_{(k)})\sim\operatorname{Beta}(k,n-k+1). $$ Its density on $0<u<1$ is $$ f_{F(X_{(k)})}(u)=\frac{n!}{(k-1)!(n-k)!}\,u^{k-1}(1-u)^{n-k}. $$

Proof. By the probability integral transform, $U_1,\ldots,U_n$ are iid uniform, and $F(X_{(k)})=U_{(k)}$ almost surely. For $0\le u\le1$, the event $U_{(k)}\le u$ occurs exactly when at least $k$ of the $n$ uniform observations are at most $u$. The number of such observations has a binomial distribution with parameters $n$ and $u$, so

P(U_{(k)}\le u)=\sum_{j=k}^{n}\binom{n}{j}u^j(1-u)^{n-j}.

Differentiating on $0<u<1$, the derivative of each summand has two parts. The positive part at index $j$ is $n\binom{n-1}{j-1}u^{j-1}(1-u)^{n-j}$; the negative part is $n\binom{n-1}{j}u^j(1-u)^{n-j-1}$. The negative part for index $j$ cancels the positive part for index $j+1$. The only surviving part is the positive part for $j=k$, giving

\frac{d}{du}P(U_{(k)}\le u) =n\binom{n-1}{k-1}u^{k-1}(1-u)^{n-k} =\frac{n!}{(k-1)!(n-k)!}u^{k-1}(1-u)^{n-k}.

This is the beta density with parameters $k$ and $n-k+1$. Since $F(X_{(k)})=U_{(k)}$ almost surely, the same distribution holds for the transformed order statistic. $\square$

Worked Example: The Transformed Second Observation

Let $n=5$ and $k=2$. Find the density of $F(X_{(2)})$, and calculate $P(F(X_{(2)})\le 1/2)$.

The theorem gives a $\operatorname{Beta}(2,4)$ distribution. Its density is

\frac{5!}{1!3!}u(1-u)^3=20u(1-u)^3,\qquad 0<u<1.

For the probability, count how many of the five transformed observations are at most $1/2$. This count is $\operatorname{Binomial}(5,1/2)$, and the second order statistic is at most $1/2$ when the count is at least 2. Therefore

P(F(X_{(2)})\le1/2) =1-P(N=0)-P(N=1) =1-\frac{1+5}{32} =\frac{26}{32} =\frac{13}{16}.

The density gives the same result: integrating $20u(1-u)^3$ from 0 to $1/2$ yields $13/16$.

Distribution-Free Probability Calculations

Write $G_k$ for the CDF of $\operatorname{Beta}(k,n-k+1)$. The theorem gives, for any real $x$,

P(X_{(k)}\le x)=P(F(X_{(k)})\le F(x))=G_k(F(x)) =\sum_{j=k}^{n}\binom{n}{j}F(x)^j[1-F(x)]^{n-j}.

The beta or binomial calculation is distribution-free in the sense that it requires only the value $F(x)$, not a particular distributional form for $F$. If $F$ is known, its value at a threshold translates the order-statistic probability into a binomial tail. If the threshold is a population $p$-quantile $q_p$ satisfying $F(q_p)=p$, then the corresponding probability depends only on $n$, $k$, and $p$.

Worked Example: An Exponential Threshold

Suppose $X_1,\ldots,X_4$ are iid exponential variables with rate $\lambda>0$. Find the probability that the third order statistic is no greater than $\ln(2)/\lambda$.

The exponential CDF is $F(x)=1-e^{-\lambda x}$ for $x\ge0$. At the stated threshold,

F\left(\frac{\ln 2}{\lambda}\right) =1-e^{-\ln 2} =\frac12.

Thus the number of observations below the threshold is $\operatorname{Binomial}(4,1/2)$. The third order statistic is below the threshold when this count is 3 or 4. Hence

P\left(X_{(3)}\le\frac{\ln 2}{\lambda}\right) =\binom43\left(\frac12\right)^4+\binom44\left(\frac12\right)^4 =\frac4{16}+\frac1{16} =\frac5{16}.

The rate $\lambda$ determines the threshold on the original scale, but it drops out once that threshold is expressed through its CDF value.

Using Order Statistics for Quantile Intervals

The same count argument gives an exact coverage probability for an interval formed from sample order statistics. Let $q_p$ be a fixed population $p$-quantile such that $F(q_p)=p$, with $0<p<1$. Assume the sample is iid from a continuous distribution. The number $N$ of observations at most $q_p$ has a $\operatorname{Binomial}(n,p)$ distribution. Since the distribution is continuous, the probability that any observation equals $q_p$ is zero. Therefore, for integers $1\le r<s\le n$,

P\bigl(X_{(r)}\le q_p\le X_{(s)}\bigr) =P(r\le N\le s-1) =\sum_{j=r}^{s-1}\binom{n}{j}p^j(1-p)^{n-j}.

This formula is exact and does not require knowing the shape or scale of $F$. It gives the coverage probability for a selected interval; choosing indices to achieve a desired coverage is a separate finite binomial calculation. Equivalently, probabilities involving $F(X_{(k)})$ can be read directly from its beta distribution.

Worked Example: An Interval for the Population Median

For an iid sample of size $n=9$ from a continuous distribution, consider the interval $[X_{(2)},X_{(8)}]$. Find its probability of containing a population median $q_{1/2}$ with $F(q_{1/2})=1/2$.

The count $N$ of sample observations at most the median is $\operatorname{Binomial}(9,1/2)$. The interval contains the median when $2\le N\le7$. The excluded cases are $N=0,1,8,9$, whose probabilities sum to

\frac{\binom90+\binom91+\binom98+\binom99}{2^9} =\frac{1+9+9+1}{512} =\frac{20}{512}.

Consequently, the coverage probability is

P\bigl(X_{(2)}\le q_{1/2}\le X_{(8)}\bigr) =1-\frac{20}{512} =\frac{492}{512} =\frac{123}{128} \approx0.9609.

Thus this interval contains the population median with probability about $96.1\%$, regardless of the continuous parent distribution.

Common Pitfall and Scope

A tempting but incorrect conclusion is that $X_{(k)}$ itself has a beta distribution. The beta result applies to $F(X_{(k)})$, which lies on the uniform probability scale. On the original scale, the distribution depends on $F$; for example, changing an exponential rate changes the scale of its order statistics, while the transformed beta law stays the same.

The assumptions are also essential. The observations must be iid, and the parent CDF must be continuous. In a discrete distribution, $F(X)$ generally is not uniform, and ties can occur. Likewise, independence alone does not give the stated beta law when the observations have different parent distributions. These cases require methods appropriate to their own assumptions.

Key takeaway: For an iid sample from a continuous parent distribution, $F(X_{(k)})\sim\operatorname{Beta}(k,n-k+1)$. On the probability scale, order-statistic events are binomial-count events, making many threshold and quantile calculations exact and distribution-free.

Check Your Understanding

Use the assumptions and results carefully in each question.

For $n=7$, what beta distribution does $F(X_{(4)})$ have?
For $n=6$, express $P(X_{(2)}\le x)$ as a binomial-tail sum in terms of $F(x)$.
Suppose $q_p$ satisfies $F(q_p)=p$. What binomial random variable determines $P(X_{(k)}\le q_p)$?
For an iid sample from a continuous distribution, explain why a flat section of $F$ does not invalidate the probability integral transform.
Why can the beta result not generally be asserted for $X_{(k)}$ itself or for a discrete parent distribution?