Independent Exponential Spacings

Why Exponential Spacings Are Useful

Order statistics are dependent: knowing that one observation is among the smallest changes what is plausible for the next one. For an iid exponential sample, however, the gaps between consecutive ordered observations have a particularly simple structure. Those gaps are independent exponential random variables, although their rates are different.

This result is more than a compact description of the ordered sample. Since each order statistic is a sum of successive gaps, its expectation and variance follow from the moments of independent exponential variables. The parameterization matters: throughout, an exponential random variable with rate $r>0$ has density $r e^{-rx}$ for $x>0$, mean $1/r$, and variance $1/r^2$.

Definition of the Spacings

Let $X_1,\ldots,X_n$ be independent and identically distributed with $X_i\sim \operatorname{Exp}(\lambda)$, where $\lambda>0$. Write their order statistics as $X_{(1)}<\cdots<X_{(n)}$; the inequalities are strict with probability one because the exponential distribution is continuous. Add the convention $X_{(0)}=0$, and define the $i$th spacing by $D_i=X_{(i)}-X_{(i-1)}$, for $i=1,\ldots,n$.

Definition: The successive spacings are $D_1=X_{(1)}$, $D_2=X_{(2)}-X_{(1)}$, and, in general, $D_i=X_{(i)}-X_{(i-1)}$. The last spacing is the gap from the second-largest observation to the largest; there is no additional spacing beyond $X_{(n)}$ in this definition.

The distinction between a spacing and an order statistic is useful: $D_i$ records just one gap, whereas $X_{(k)}$ is the location of the $k$th ordered observation. The two are related by $X_{(k)}=D_1+\cdots+D_k$.

Worked Example: Listing the Rates for Four Observations

Suppose $n=4$ and the observations have rate $\lambda=2$. The spacing rates, as we will prove below, are $4\lambda,3\lambda,2\lambda,\lambda$. Thus $D_1,D_2,D_3,D_4$ are independent exponentials with respective rates $8,6,4,2$.

Their respective means are the reciprocals of those rates: $E[D_1]=1/8$, $E[D_2]=1/6$, $E[D_3]=1/4$, and $E[D_4]=1/2$. The first gap tends to be smaller because it is the minimum of four observations. The final gap has rate $2$, not $8$, because only one observation remains to be the largest.

For instance, $X_{(2)}=D_1+D_2$, so $E[X_{(2)}]=1/8+1/6=7/24$, and independence gives $\operatorname{Var}(X_{(2)})=1/8^2+1/6^2=1/64+1/36=25/576$. The fact that these are sums of independent gaps is what makes both calculations direct.

The Joint Density and Independence

The joint density of the ordered sample is a standard order-statistic result. In this case it can be written explicitly: for $0<x_1<\cdots<x_n$, it is $n!\lambda^n\exp\{-\lambda\sum_{i=1}^n x_i\}$, and it is zero outside this region. The factor $n!$ accounts for the $n!$ possible label orderings of distinct observations.

Theorem: If $X_1,\ldots,X_n$ are iid $\operatorname{Exp}(\lambda)$, with $\lambda>0$, then $D_1,\ldots,D_n$ are mutually independent and $D_i\sim\operatorname{Exp}((n-i+1)\lambda)$ for $i=1,\ldots,n$.

To establish the result, transform from ordered values $x_1,\ldots,x_n$ to gaps $d_1,\ldots,d_n$. The transformation and its inverse are $d_i=x_i-x_{i-1}$, with $x_0=0$, and $x_i=\sum_{j=1}^i d_j$. The ordered region $0<x_1<\cdots<x_n$ becomes the positive orthant $d_1>0,\ldots,d_n>0$. The Jacobian determinant of the inverse transformation is 1: its matrix is lower triangular with every diagonal entry equal to 1.

Express the ordered values through the gaps.
For each $i$, $x_i=\sum_{j=1}^i d_j$, so $\sum_{i=1}^n x_i=\sum_{j=1}^n(n-j+1)d_j$. The term $d_j$ appears in $x_j,x_{j+1},\ldots,x_n$, which is $n-j+1$ terms.

Apply the change-of-variables formula.
The transformed density is $n!\lambda^n\exp\{-\lambda\sum_{j=1}^n(n-j+1)d_j\}$ on the positive orthant, because the Jacobian determinant is 1.

Factor the density.
Since $n!\lambda^n=\prod_{j=1}^n(n-j+1)\lambda$, the joint density is a product of exponential densities.

In full, the factorization is

f_{D_1,\ldots,D_n}(d_1,\ldots,d_n) =\prod_{j=1}^n\left[(n-j+1)\lambda \exp\{-(n-j+1)\lambda d_j\}\right], \qquad d_1,\ldots,d_n>0.

Each factor integrates to 1 over its positive coordinate, and is the density of an exponential random variable with rate $(n-j+1)\lambda$. A joint density that factors into these marginal densities establishes mutual independence. Thus the first spacing has rate $n\lambda$, the second has rate $(n-1)\lambda$, and the last has rate $\lambda$. This proves the theorem. $\square$

Moments of Exponential Order Statistics

An exponential random variable with rate $r$ has mean $1/r$ and variance $1/r^2$. Since $X_{(k)}=D_1+\cdots+D_k$, linearity of expectation and independence of the spacings give the order-statistic moment formulas below. Define $H_m=\sum_{j=1}^m 1/j$ and $H_m^{(2)}=\sum_{j=1}^m 1/j^2$, with $H_0=H_0^{(2)}=0$.

Result: For $1\leq k\leq n$, $$ E[X_{(k)}]=\frac{1}{\lambda}\sum_{j=n-k+1}^n\frac1j =\frac{H_n-H_{n-k}}{\lambda}, \qquad \operatorname{Var}(X_{(k)})=\frac{1}{\lambda^2}\sum_{j=n-k+1}^n\frac1{j^2} =\frac{H_n^{(2)}-H_{n-k}^{(2)}}{\lambda^2}. $$

For the expectation, add the means of the first $k$ spacings:

E[X_{(k)}] =\sum_{i=1}^k E[D_i] =\sum_{i=1}^k\frac{1}{(n-i+1)\lambda} =\frac{1}{\lambda}\sum_{j=n-k+1}^n\frac1j.

The final equality uses the index change $j=n-i+1$. For the variance, the same spacings are mutually independent, so the variance of their sum is the sum of their variances:

\operatorname{Var}(X_{(k)}) =\sum_{i=1}^k\operatorname{Var}(D_i) =\sum_{i=1}^k\frac{1}{(n-i+1)^2\lambda^2} =\frac{1}{\lambda^2}\sum_{j=n-k+1}^n\frac1{j^2}.

The index changes yield the harmonic-number expressions in the result. These formulas require no approximation and hold for every positive integer $n$, every $1\leq k\leq n$, and every $\lambda>0$.

Corollary: If $1\leq r\leq s\leq n$, then $\operatorname{Cov}(X_{(r)},X_{(s)})=\operatorname{Var}(X_{(r)})$. Indeed, $X_{(s)}=X_{(r)}+\sum_{i=r+1}^sD_i$, and that remaining sum is independent of $X_{(r)}$.

For the covariance, bilinearity gives $\operatorname{Cov}(X_{(r)},X_{(s)})=\operatorname{Cov}(X_{(r)},X_{(r)})+\operatorname{Cov}(X_{(r)},\sum_{i=r+1}^sD_i)$. The first term is $\operatorname{Var}(X_{(r)})$. The second is zero because $X_{(r)}$ is a function of $D_1,\ldots,D_r$, and the later spacings are independent of these. This proves the stated identity. $\square$

Worked Example: Mean and Variance of the Third Order Statistic

Let $n=5$ and $\lambda=3/2$. The first three spacings have rates $15/2$, $6$, and $9/2$, respectively. Therefore $X_{(3)}=D_1+D_2+D_3$, with these three independent terms.

Its mean is

E[X_{(3)}] =\frac{1}{15/2}+\frac16+\frac{1}{9/2} =\frac{2}{15}+\frac16+\frac29 =\frac{12+15+20}{90} =\frac{47}{90}.

Its variance is

\operatorname{Var}(X_{(3)}) =\frac{1}{(15/2)^2}+\frac1{6^2}+\frac{1}{(9/2)^2} =\frac4{225}+\frac1{36}+\frac4{81} =\frac{144+225+400}{8100} =\frac{769}{8100}.

The harmonic formulas provide the same check: $E[X_{(3)}]=(1/1.5)(1/3+1/4+1/5)=\frac23\cdot\frac{47}{60}=\frac{47}{90}$, and $\operatorname{Var}(X_{(3)})=(1/1.5^2)(1/9+1/16+1/25)=\frac49\cdot\frac{769}{3600}=\frac{769}{8100}$.

The direct spacing rates listed above are correct: $D_1$ rate $5(3/2)=15/2$, $D_2$ rate $4(3/2)=6$, and $D_3$ rate $3(3/2)=9/2$. Their reciprocal squares sum to $4/225+1/36+4/81=144/8100+225/8100+400/8100=769/8100$, agreeing with the harmonic check $(4/9)(1/9+1/16+1/25)=(4/9)(769/3600)=769/8100$.

Worked Example: Covariance of Two Order Statistics

Suppose $n=4$, $\lambda=1$, and consider $X_{(2)}$ and $X_{(4)}$. The covariance identity says their covariance is the variance of $X_{(2)}$. Verify it directly from spacings.

Here $X_{(2)}=D_1+D_2$, where the rates are 4 and 3, and $X_{(4)}=D_1+D_2+D_3+D_4$, where the remaining rates are 2 and 1. Consequently, $\operatorname{Cov}(X_{(2)},X_{(4)})=\operatorname{Var}(D_1)+\operatorname{Var}(D_2)$; every covariance between distinct spacings is zero.

The value is $\frac1{4^2}+\frac1{3^2}=\frac1{16}+\frac19=\frac{25}{144}$. Meanwhile, $\operatorname{Var}(X_{(2)})$ is the same sum, so the direct calculation confirms the identity. The mean of $X_{(2)}$, as an additional check, is $1/4+1/3=7/12$.

Why the Rates Decrease

The rate of $D_i$ is $(n-i+1)\lambda$, and its mean is $1/((n-i+1)\lambda)$. As $i$ increases, fewer observations remain to appear above the already observed order statistics. The early gaps are compressed near the minimum, while later gaps are typically larger. The rates therefore decrease, and the spacing means increase.

This is a structural property of the exponential distribution, connected to its constant hazard rate. The density calculation gives the precise reason here: after transforming to gaps, the exponent separates into a sum whose coefficient on $d_i$ is $(n-i+1)\lambda$. No claim that arbitrary iid distributions have independent spacings follows from this result.

Common Mistake

A frequent error is to list the spacing rates in the order $\lambda,2\lambda,\ldots,n\lambda$, or to mistake a rate for a mean. The minimum $D_1=X_{(1)}$ has survival probability $P(D_1>t)=P(X_1>t,\ldots,X_n>t)=e^{-n\lambda t}$, so its rate must be $n\lambda$, not $\lambda$. The corresponding mean is $1/(n\lambda)$. At the other end, $D_n$ has rate $\lambda$ and mean $1/\lambda$.

A second related error is to calculate the variance of $X_{(k)}$ by adding the standard deviations of its spacings. Variances add for independent terms; standard deviations generally do not. The correct calculation is $\operatorname{Var}(X_{(k)})=\sum_{i=1}^k1/((n-i+1)^2\lambda^2)$.

Key takeaway: In an iid exponential sample with rate $\lambda$, the successive spacings are independent with rates $n\lambda,(n-1)\lambda,\ldots,\lambda$. Each order statistic is a partial sum of those gaps, so its mean and variance are sums of the corresponding reciprocal rates and squared reciprocal rates.

Check Your Understanding

Use the spacing rates and the fact that independent variances add to answer these questions.

For $n=3$ and rate $\lambda$, what are the distributions of $D_1,D_2,D_3$?
For $n=4$ and $\lambda=2$, calculate $E[X_{(3)}]$ as a sum of spacing means.
For $n=5$, express $\operatorname{Var}(X_{(2)})$ in terms of $\lambda$.
Explain why $D_1$ has rate $n\lambda$, rather than $\lambda$.
If $r\leq s$, why is $\operatorname{Cov}(X_{(r)},X_{(s)})=\operatorname{Var}(X_{(r)})$?