Joint Densities of Selected Order Statistics

Why Two Ranks Require Three Counts

A single order statistic is located by counting how many observations fall below its value. Two selected ranks require a more detailed accounting. If \(X_{(r)}=x\) and \(X_{(s)}=y\), where \(r<s\), then there are \(r-1\) observations below \(x\), \(s-r-1\) between \(x\) and \(y\), and \(n-s\) above \(y\). The two observations at the selected ranks contribute one observation at each selected value.

That partition of the sample is the key to the joint density. The probabilities of the three regions become powers of the CDF, the CDF difference, and the upper-tail probability. A combinatorial coefficient counts the ways to assign the \(n\) labeled observations to those regions and the two selected ranks.

Setup: Let \(X_1,\ldots,X_n\) be iid random variables with continuous distribution function \(F\) and density \(f\). Write \(X_{(1)}\leq\cdots\leq X_{(n)}\) for their order statistics. For integers \(1\leq r<s\leq n\), the pair \((X_{(r)},X_{(s)})\) has support on \(x<y\), where the sample can take values.

Joint Density of Two Selected Order Statistics

The formula below applies to any two distinct selected ranks. Its factorials record the numbers of observations in each of the three regions. The convention \(0!=1\) handles a region that contains no observations.

Theorem: Under the setup above, a joint density of \((X_{(r)},X_{(s)})\) is

f_{X_{(r)},X_{(s)}}(x,y) = \frac{n!}{(r-1)!(s-r-1)!(n-s)!} [F(x)]^{r-1} [F(y)-F(x)]^{s-r-1} [1-F(y)]^{n-s}f(x)f(y), \qquad x<y.

The density is zero for \(x\geq y\).

The factors have direct interpretations. An observation is below \(x\) with probability \(F(x)\); it is between \(x\) and \(y\) with probability \(F(y)-F(x)\); and it is above \(y\) with probability \(1-F(y)\). The selected observations themselves contribute the density factors \(f(x)\) and \(f(y)\).

Derivation from the Ordered Sample

Because the common distribution is continuous, ties occur with probability zero. For example, for any two different indices \(i\) and \(j\), independence and continuity give \(P(X_i=X_j)=0\); taking a finite union over pairs shows that all sample values are distinct with probability one.

On the region \(z_1<z_2<\cdots<z_n\), the joint density of the ordered sample is

f_{X_{(1)},\ldots,X_{(n)}}(z_1,\ldots,z_n) = n!\prod_{j=1}^{n}f(z_j).

Indeed, each strictly ordered vector of values can arise from \(n!\) permutations of the labeled observations. Each permutation has joint density \(\prod_{j=1}^n f(z_j)\), so summing those contributions gives the displayed density.

Now fix \(x<y\), and integrate out the ordered observations other than \(z_r=x\) and \(z_s=y\). The \(r-1\) values below \(x\) must appear in increasing order. Their product-density integral is

\int_{-\infty<z_1<\cdots<z_{r-1}<x} \prod_{j=1}^{r-1} f(z_j)\,dz_1\cdots dz_{r-1} = \frac{[F(x)]^{r-1}}{(r-1)!}.

To see the factorial, consider the full region where \(r-1\) coordinates all lie below \(x\). It is a cube, apart from its measure-zero ties, and splits into \((r-1)!\) orderings. The product integrand is symmetric, so each ordering has the same integral. The integral over the whole cube is \([F(x)]^{r-1}\), giving the result after division by \((r-1)!\). The same argument applies to any interval: \(m\) ordered coordinates in an interval with probability mass \(q\) contribute \(q^m/m!\).

Applying that argument to the \(s-r-1\) values between \(x\) and \(y\) gives \([F(y)-F(x)]^{s-r-1}/(s-r-1)!\). Applying it to the \(n-s\) values above \(y\) gives \([1-F(y)]^{n-s}/(n-s)!\). Multiplication by the ordered-sample factor \(n!\) and the two fixed-coordinate densities \(f(x)f(y)\) proves the theorem. The integrations are valid for a density \(f\), since the product density is nonnegative and Tonelli's theorem permits the iterated integrations. \(\square\)

Worked Examples

Worked Example: Two Ranks from a Uniform Sample

Let \(X_1,\ldots,X_5\) be iid Uniform\((0,1)\), and find the joint density of \(X_{(2)}\) and \(X_{(4)}\).

Here \(n=5\), \(r=2\), and \(s=4\). On \(0<x<y<1\), the density and CDF are \(f(t)=1\) and \(F(t)=t\). The numbers below, between, and above the selected values are \(1\), \(1\), and \(1\), respectively. Thus the coefficient is

\frac{5!}{1!1!1!}=120,

and the joint density is

f_{X_{(2)},X_{(4)}}(x,y) = 120x(y-x)(1-y), \qquad 0<x<y<1.

As a check, integrate the density. For fixed \(x\), put \(L=1-x\) and \(t=y-x\). Then

\int_x^1 (y-x)(1-y)\,dy = \int_0^L t(L-t)\,dt = \left[\frac{Lt^2}{2}-\frac{t^3}{3}\right]_0^L = \frac{L^3}{6}.

Therefore the total integral is

120\int_0^1 x\frac{(1-x)^3}{6}\,dx = 20\int_0^1 x(1-x)^3\,dx = 20\left(\frac12-\frac33+\frac34-\frac15\right) = 20\left(\frac1{20}\right) = 1.

Worked Example: Two Ranks from an Exponential Sample

Let \(X_1,\ldots,X_4\) be iid exponential random variables with rate \(\lambda>0\). Find the joint density of \(X_{(1)}\) and \(X_{(3)}\).

For \(t>0\), \(f(t)=\lambda e^{-\lambda t}\) and \(F(t)=1-e^{-\lambda t}\). Here \(n=4\), \(r=1\), \(s=3\), so there are zero observations below \(x\), one between \(x\) and \(y\), and one above \(y\). The coefficient is \(4!/(0!1!1!)=24\). Substituting into the general formula gives

\begin{aligned} f_{X_{(1)},X_{(3)}}(x,y) &=24[F(y)-F(x)][1-F(y)]f(x)f(y)\\ &=24(e^{-\lambda x}-e^{-\lambda y})e^{-\lambda y} (\lambda e^{-\lambda x})(\lambda e^{-\lambda y})\\ &=24\lambda^2 \left(e^{-2\lambda x-2\lambda y}-e^{-\lambda x-3\lambda y}\right), \qquad 0<x<y. \end{aligned}

The first exponential term comes from multiplying \(e^{-\lambda x}\) by the three factors \(e^{-\lambda y}\). The second comes from multiplying \(e^{-\lambda y}\) by those same three factors. Both terms are needed: they represent the difference \(F(y)-F(x)\), not just the probability of being below \(y\).

Worked Example: Two Ranks from a Normal Sample

Let \(X_1,\ldots,X_6\) be iid \(N(0,1)\), and find the joint density of \(X_{(2)}\) and \(X_{(5)}\). Write \(\phi\) and \(\Phi\) for the standard normal density and CDF.

There is one observation below \(x\), two between \(x\) and \(y\), and one above \(y\). The coefficient is

\frac{6!}{1!2!1!}=360.

Since \(f=\phi\) and \(F=\Phi\), the density is

f_{X_{(2)},X_{(5)}}(x,y) = 360\Phi(x)[\Phi(y)-\Phi(x)]^2[1-\Phi(y)]\phi(x)\phi(y), \qquad x<y.

For a numerical check at \(x=0\) and \(y=1\), use \(\Phi(0)=0.5\), \(\Phi(1)\approx0.8413447\), \(\phi(0)\approx0.3989423\), and \(\phi(1)\approx0.2419707\). The density there is

360(0.5)(0.3413447)^2(0.1586553)(0.3989423)(0.2419707) \approx 0.321.

This is a density value at a point, not the probability that the two order statistics equal exactly \(0\) and \(1\). Probabilities are obtained by integrating the joint density over a region in the \((x,y)\)-plane.

A Marginal Check

Integrating out \(X_{(s)}\) must recover the density of \(X_{(r)}\). This provides a useful algebraic check on the powers, factorials, and support in the joint formula.

Result: Integrating the joint density over \(y>x\) gives

f_{X_{(r)}}(x) = \frac{n!}{(r-1)!(n-r)!} [F(x)]^{r-1}[1-F(x)]^{n-r}f(x).

To verify this, set \(a=F(x)\) and integrate the joint density with respect to \(y\). For an absolutely continuous CDF, the substitution \(u=F(y)\) in an integral against \(f(y)\,dy\) is valid; equivalently, it follows by applying the chain rule to an antiderivative in \(u\). Thus the part of the integral depending on \(y\) is

\begin{aligned} \int_x^\infty [F(y)-a]^{s-r-1}[1-F(y)]^{n-s} f(y)\,dy &=(1-a)^{n-r} \int_0^1 t^{s-r-1}(1-t)^{n-s}\,dt\\ &=(1-a)^{n-r} \frac{(s-r-1)!(n-s)!}{(n-r)!}. \end{aligned}

In the first line, \(t=(u-a)/(1-a)\). The last integral is the integer-parameter beta integral; repeated integration by parts gives \(\int_0^1 t^p(1-t)^q\,dt=p!q!/(p+q+1)!\) for nonnegative integers \(p,q\). Multiplying by the joint-density coefficient and the remaining factors gives

\frac{n!}{(r-1)!(s-r-1)!(n-s)!} \cdot \frac{(s-r-1)!(n-s)!}{(n-r)!} [F(x)]^{r-1}[1-F(x)]^{n-r}f(x),

which is exactly the stated marginal density. This confirms that the joint density integrates over the second coordinate to the correct one-dimensional density. \(\square\)

Common Pitfall: Counting the Regions Incorrectly

A frequent error is to use \(s-r\) as the exponent on \(F(y)-F(x)\). That count includes the observation at rank \(r\), which is at \(x\), rather than strictly between the selected values. The number strictly between the selected ranks is \(s-r-1\). Likewise, the \(r-1\) observations below \(x\) and the \(n-s\) above \(y\) are counts of unselected observations.

Another mistake is to read \(f_{X_{(r)},X_{(s)}}(x,y)\) as a point probability. With continuous observations, the probability of either order statistic taking one exact value is zero. A joint density gives probabilities only after integration over a region. For a small rectangle around \((x,y)\) with \(x<y\), its probability is approximately the density at \((x,y)\) times the rectangle's area, provided the density is continuous there.

Key takeaway: For two selected ranks \(r<s\), count \(r-1\) observations below the first value, \(s-r-1\) between the values, and \(n-s\) above the second. Those three counts determine the powers in the joint density; the factorial coefficient accounts for all assignments of the labeled observations.

Check Your Understanding

Use the region-counting argument and the joint-density formula to answer these questions.

For \(n=7\), \(r=2\), and \(s=6\), how many observations lie below, between, and above the two selected values?
For a Uniform\((0,1)\) sample of size \(6\), write the joint density of \(X_{(2)}\) and \(X_{(5)}\), including its support.
Explain why the joint density contains \(f(x)f(y)\) in addition to the three powers involving \(F\).
For \(n=5\), \(r=1\), and \(s=4\), calculate the combinatorial coefficient and the exponent on \(F(y)-F(x)\).
Why does a value of the joint density at one point not give the probability that the order statistics equal those two values?