Cross-Moments of Two Order Statistics

Why Cross-Moments Matter

Two order statistics from the same sample are linked: if one observation is unusually small, for example, that can affect both the minimum and the second-smallest value. Their individual distributions describe each one separately, but not how they vary together. A cross-moment, $E[X_{(r)}X_{(s)}]$, captures part of that joint behavior and leads directly to their covariance and correlation.

The key calculation is to integrate the product $xy$ against the joint density of the selected order statistics. The integration region is triangular because $X_{(r)}\leq X_{(s)}$. The formula below applies to an iid continuous sample; its moment consequences require the corresponding moments to exist.

Joint Density of Two Selected Order Statistics

Definition: Let $X_1,\ldots,X_n$ be iid random variables with an absolutely continuous distribution, density $f$, and CDF $F$. Write their ordered values as $X_{(1)}\leq\cdots\leq X_{(n)}$. For integers $1\leq r<s\leq n$, the joint density of $X_{(r)}$ and $X_{(s)}$ is $$ f_{r,s}(x,y)=\frac{n!}{(r-1)!(s-r-1)!(n-s)!} f(x)f(y)F(x)^{r-1}\bigl(F(y)-F(x)\bigr)^{s-r-1} \bigl(1-F(y)\bigr)^{n-s},\qquad x<y, $$ and is zero for $x\geq y$.

The factors have a rank-count interpretation. There are $r-1$ observations below $x$, $s-r-1$ between $x$ and $y$, and $n-s$ above $y$; the observations at the two selected ranks contribute $f(x)f(y)$. The factorial coefficient counts the ways to assign sample labels to these roles.

Theorem: The displayed expression is a joint density for $X_{(r)},X_{(s)}$.

Proof. Since the sample is absolutely continuous, ties have probability zero. On the region $z_1<\cdots<z_n$, the joint density of the fully ordered sample is $$ n!\prod_{i=1}^{n}f(z_i). $$ Indeed, each strictly ordered vector can arise from $n!$ permutations of the labelled sample, and each permutation has the same product density. Integrating out all ordered coordinates except $z_r=x$ and $z_s=y$ gives the joint density of the selected pair.

For $k$ ordered coordinates below $x$, integrating their product density over their ordered region gives $F(x)^k/k!$. To see this, the integral over the entire $k$-dimensional product region is $F(x)^k$; the $k!$ possible strict orderings have equal integrals by symmetry, and their boundaries have measure zero. The same argument gives $(F(y)-F(x))^k/k!$ for coordinates between $x$ and $y$, and $(1-F(y))^k/k!$ for coordinates above $y$. Thus integrating out the remaining coordinates gives $$ \frac{n!}{(r-1)!(s-r-1)!(n-s)!}f(x)f(y)F(x)^{r-1} \bigl(F(y)-F(x)\bigr)^{s-r-1}\bigl(1-F(y)\bigr)^{n-s}. $$ This density is supported on $x<y$, as required by the ordering. $\square$

Product Moments, Covariance, and Correlation

The marginal density of $X_{(k)}$, useful for calculating its mean and variance, is $$ f_{(k)}(x)=\frac{n!}{(k-1)!(n-k)!}f(x)F(x)^{k-1}\bigl(1-F(x)\bigr)^{n-k}. $$ Integrating the joint density gives the cross-moment. The required integrability condition is important: if $E[X_{(r)}^2]$ and $E[X_{(s)}^2]$ are finite, then $E[|X_{(r)}X_{(s)}|]$ is finite by the Cauchy–Schwarz inequality.

Theorem: If $E[X_{(r)}^2]$ and $E[X_{(s)}^2]$ are finite, then $$ E[X_{(r)}X_{(s)}]=\int_{-\infty}^{\infty}\int_x^\infty xy\,f_{r,s}(x,y)\,dy\,dx, \qquad \operatorname{Cov}(X_{(r)},X_{(s)}) =E[X_{(r)}X_{(s)}]-E[X_{(r)}]E[X_{(s)}]. $$ If both variances are positive, their correlation is $$ \operatorname{Corr}(X_{(r)},X_{(s)}) =\frac{E[X_{(r)}X_{(s)}]-E[X_{(r)}]E[X_{(s)}]} {\sqrt{\operatorname{Var}(X_{(r)})\operatorname{Var}(X_{(s)})}}. $$

Proof. The joint density formula gives the expectation of any integrable function $g(X_{(r)},X_{(s)})$ by integrating $g(x,y)f_{r,s}(x,y)$ over its support. Take $g(x,y)=xy$. The second-moment assumptions ensure absolute integrability because $$ E[|X_{(r)}X_{(s)}|]\leq \sqrt{E[X_{(r)}^2]E[X_{(s)}^2]}<\infty. $$ The covariance identity follows from its definition, and dividing by the product of the positive standard deviations gives the correlation formula. $\square$

Worked Example: Minimum and Maximum of Two Exponential Observations

Let $X_1,X_2$ be iid exponential random variables with rate $1$, so $f(x)=e^{-x}$ for $x>0$. Find the product moment, covariance, and correlation of $X_{(1)}$ and $X_{(2)}$.

For $n=2,r=1,s=2$, the joint density is $$ f_{1,2}(x,y)=2e^{-x-y},\qquad 0<x<y. $$ Set $u=x$ and $v=y-x$, so $u,v>0$, $y=u+v$, and the Jacobian is $1$. The transformed density is $2e^{-2u-v}$. The integrals $\int_0^\infty t^k e^{-at}\,dt=k!/a^{k+1}$ follow by repeated integration by parts. Therefore $$ E[X_{(1)}X_{(2)}] =2\int_0^\infty\int_0^\infty u(u+v)e^{-2u-v}\,dv\,du =2\left(\frac14+\frac14\right)=1. $$ Here the two terms inside parentheses are respectively $\left(\int u^2e^{-2u}du\right)\left(\int e^{-v}dv\right)=1/4$ and $\left(\int ue^{-2u}du\right)\left(\int ve^{-v}dv\right)=1/4$.

The same density, factored in $u,v$, gives $E[X_{(1)}]=E[u]=1/2$ and $E[X_{(2)}]=E[u+v]=1/2+1=3/2$. Also, $$ E[X_{(1)}^2]=\frac12,\qquad E[X_{(2)}^2]=E[(u+v)^2]=\frac12+2\left(\frac12\right)(1)+2=\frac72. $$ Thus the variances are $1/2-(1/2)^2=1/4$ and $7/2-(3/2)^2=5/4$. The covariance is $1-(1/2)(3/2)=1/4$, and the correlation is $$ \frac{1/4}{\sqrt{(1/4)(5/4)}}=\frac{1}{\sqrt{5}}\approx 0.4472. $$

Worked Example: The First Two Order Statistics of Three Exponentials

Let $X_1,X_2,X_3$ be iid exponential with rate $1$. Calculate the product moment, covariance, and correlation of $X_{(1)}$ and $X_{(2)}$.

The joint density for $r=1,s=2,n=3$ is $6e^{-x-2y}$ on $0<x<y$. Under $u=x,\ v=y-x$, this becomes $6e^{-3u-2v}$. The density factors into exponential densities with rates $3$ and $2$. Consequently, $E[u]=1/3$, $E[v]=1/2$, $E[u^2]=2/9$, and $E[v^2]=1/2$. Since $X_{(1)}=u$ and $X_{(2)}=u+v$, $$ E[X_{(1)}X_{(2)}]=E[u^2]+E[u]E[v] =\frac29+\frac16=\frac7{18}. $$ The means are $1/3$ and $1/3+1/2=5/6$. The variances are $$ \operatorname{Var}(X_{(1)})=\frac29-\left(\frac13\right)^2=\frac19, \qquad \operatorname{Var}(X_{(2)})=\frac{19}{18}-\left(\frac56\right)^2=\frac{13}{36}, $$ where $E[X_{(2)}^2]=E[(u+v)^2]=2/9+2(1/3)(1/2)+1/2=19/18$. Hence $$ \operatorname{Cov}(X_{(1)},X_{(2)})=\frac7{18}-\frac13\frac56=\frac19, \qquad \operatorname{Corr}(X_{(1)},X_{(2)})= \frac{1/9}{\sqrt{(1/9)(13/36)}}=\frac{2}{\sqrt{13}}\approx0.5547. $$

Worked Example: Two Order Statistics from a Beta(2,1) Sample

Suppose $X_1,X_2$ are iid with density $f(x)=2x$ for $0<x<1$, and $F(x)=x^2$. For the minimum and maximum, the joint density is $2f(x)f(y)=8xy$ on $0<x<y<1$. Their product moment is $$ E[X_{(1)}X_{(2)}] =\int_0^1\int_0^y xy(8xy)\,dx\,dy =8\int_0^1 y^2\left(\frac{y^3}{3}\right)dy =\frac49. $$ The marginal densities are $f_{(1)}(x)=4x(1-x^2)$ and $f_{(2)}(x)=4x^3$, for $0<x<1$. Direct integration gives $$ E[X_{(1)}]=4\left(\frac13-\frac15\right)=\frac8{15}, \quad E[X_{(2)}]=\frac45, \quad E[X_{(1)}^2]=\frac13, \quad E[X_{(2)}^2]=\frac23. $$ Thus the variances are $1/3-64/225=11/225$ and $2/3-16/25=2/75$. It follows that $$ \operatorname{Cov}(X_{(1)},X_{(2)})=\frac49-\frac8{15}\frac45=\frac4{225}, \qquad \operatorname{Corr}(X_{(1)},X_{(2)})= \frac{4/225}{\sqrt{(11/225)(2/75)}}=\frac4{\sqrt{66}}\approx0.4924. $$ The positive covariance is consistent with the calculation: a sample whose two observations are both relatively large tends to have both a larger minimum and a larger maximum.

Interpretation and a Common Pitfall

The cross-moment is not itself a measure of association with a fixed zero point: shifting either variable changes $E[X_{(r)}X_{(s)}]$. Covariance removes the product of the means, while correlation also scales by the marginal standard deviations. Correlation is therefore unit-free, provided both variances are positive. The joint-density integral is especially useful when the distribution has a simple density and the order-statistic region is manageable.

Mistake: treating the order statistics as independent. One might calculate $E[X_{(r)}X_{(s)}]$ as $E[X_{(r)}]E[X_{(s)}]$ simply because the original observations are independent. This is incorrect: the ordered values are functions of the same sample, and their ranks impose a dependence. The exponential example gives $E[X_{(1)}X_{(2)}]=1$, whereas the product of their means is $3/4$.
Correction: Integrate $xy$ against the joint density, or establish independence separately if a problem provides a valid reason. Independence of the original observations alone does not establish independence of their order statistics.
Mistake: ignoring the triangular support. The pair density is zero unless $x<y$. Integrating over the whole plane, or reversing the limits without changing the region, does not calculate the required expectation.

Key takeaway: For $r<s$, calculate $E[X_{(r)}X_{(s)}]$ over $x<y$ using the joint order-statistic density. Then use the marginal moments to obtain covariance and, when both variances are positive, correlation.

Check Your Understanding

Try these questions using the definitions and formulas above.

In the joint density of $X_{(r)}$ and $X_{(s)}$, what do the powers of $F(x)$, $F(y)-F(x)$, and $1-F(y)$ count?
Why do finite second moments of both selected order statistics guarantee that their product moment exists?
For $X_1,X_2$ iid exponential with rate $1$, compare the product moment with the product of the means. What does the comparison say about independence?
What additional condition, beyond existence of the product moment, is needed to define the correlation?
For a continuous sample, explain why the joint density of $X_{(r)},X_{(s)}$ is zero when $x\geq y$.

Cross-Moments of Two Order Statistics

What You'll Learn