Order Statistics from a Normal Sample

Normal Samples and Their Order Statistics

A normal sample is easy to describe one observation at a time, but the smallest, largest, or middle observation has a different distribution from any single observation. Its distribution reflects the chance that the other sample values fall on either side of it. The resulting formulas are exact, although their integrals do not always simplify to elementary functions.

Let $X_1,\ldots,X_n$ be independent and identically distributed as $N(\mu,\sigma^2)$, where $\sigma>0$. Write their ordered values as $X_{(1)}\leq \cdots\leq X_{(n)}$. The standard normal density and distribution function are denoted by $\phi$ and $\Phi$:

\phi(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2}, \qquad \Phi(z)=\int_{-\infty}^{z}\phi(t)\,dt.

For an observation $x$, its standardized value is $z=(x-\mu)/\sigma$. The normal density and distribution function at $x$ are therefore $f(x)=\frac{1}{\sigma}\phi((x-\mu)/\sigma)$ and $F(x)=\Phi((x-\mu)/\sigma)$.

Definition: For $k=1,\ldots,n$, $X_{(k)}$ is the $k$th smallest observation in the sample. Thus $X_{(1)}$ is the sample minimum, $X_{(n)}$ is the sample maximum, and ranks between them are central or intermediate order statistics.

The Exact Density and Moment Integrals

At a value $x$, the event that the $k$th order statistic falls near $x$ requires one observation near $x$, exactly $k-1$ observations below $x$, and $n-k$ above $x$. The number of ways to assign those roles accounts for the combinatorial factor in its density.

Theorem: If $X_1,\ldots,X_n$ are IID with a continuous distribution having density $f$ and distribution function $F$, then the density of $X_{(k)}$ is $$ f_{(k)}(x)=\frac{n!}{(k-1)!(n-k)!} [F(x)]^{k-1}[1-F(x)]^{n-k}f(x). $$ For an IID $N(\mu,\sigma^2)$ sample, substitute $F(x)=\Phi((x-\mu)/\sigma)$ and $f(x)=\phi((x-\mu)/\sigma)/\sigma$.

To justify the density, consider a small interval $(x,x+h]$. The leading-order way for $X_{(k)}$ to fall there is for one observation to lie in the interval, $k-1$ observations to be below $x$, and $n-k$ to be above $x+h$. Independence gives probability \[ \frac{n!}{(k-1)!(n-k)!} F(x)^{k-1}\{1-F(x+h)\}^{n-k}\{F(x+h)-F(x)\}. \] As $h$ decreases to zero, continuity of $F$ gives $F(x+h)\to F(x)$, and differentiability with density $f$ gives $\{F(x+h)-F(x)\}/h\to f(x)$. Configurations with two or more observations in the interval have probability of smaller order than $h$: since $F(x+h)-F(x)=O(h)$, their probability is $O(h^2)$. Dividing by $h$ and taking the limit yields the stated density. This argument applies at points where $f$ is continuous, as it is for a normal distribution. $\square$

Consequently, whenever $g(X_{(k)})$ is integrable, its expectation is the density integral

E[g(X_{(k)})] = \frac{n!}{(k-1)!(n-k)!} \int_{-\infty}^{\infty} g(x)F(x)^{k-1}[1-F(x)]^{n-k}f(x)\,dx.

For a normal sample, the substitution $z=(x-\mu)/\sigma$ gives a particularly convenient form:

E[g(X_{(k)})] = \frac{n!}{(k-1)!(n-k)!} \int_{-\infty}^{\infty} g(\mu+\sigma z)\, \Phi(z)^{k-1}[1-\Phi(z)]^{n-k}\phi(z)\,dz.

Normal order statistics have finite moments of every positive integer order, so this formula applies, in particular, to $g(x)=x$ and $g(x)=x^2$. The same standardization also gives $X_{(k)}=\mu+\sigma Z_{(k)}$ when the $Z_i=(X_i-\mu)/\sigma$ are ordered. Thus $E[X_{(k)}]=\mu+\sigma E[Z_{(k)}]$, and $\operatorname{Var}(X_{(k)})=\sigma^2\operatorname{Var}(Z_{(k)})$.

Sample Minimum and Maximum

For the minimum and maximum, the general formula simplifies because one of the two powers is zero. The minimum density weights a possible value by the chance that all other observations exceed it; the maximum density weights it by the chance that all others fall below it.

f_{(1)}(x)=n[1-F(x)]^{n-1}f(x), \qquad f_{(n)}(x)=nF(x)^{n-1}f(x).

The corresponding distribution functions also have direct event interpretations. The maximum is at most $x$ exactly when every observation is at most $x$. The minimum exceeds $x$ exactly when every observation exceeds $x$.

P(X_{(n)}\leq x)=F(x)^n, \qquad P(X_{(1)}>x)=[1-F(x)]^n.

For a normal sample, these become $\Phi((x-\mu)/\sigma)^n$ and $[1-\Phi((x-\mu)/\sigma)]^n$, respectively. These event formulas are often simpler than integrating a density.

Worked Example: A Maximum Below One Standard Deviation Above the Mean

Let $X_1,\ldots,X_4$ be IID $N(\mu,\sigma^2)$. Find the exact probability that the sample maximum is no greater than $\mu+\sigma$, and give its numerical value.

The event $X_{(4)}\leq \mu+\sigma$ means that all four observations are at most $\mu+\sigma$. Standardizing the threshold gives $(\mu+\sigma-\mu)/\sigma=1$. Independence therefore yields

P(X_{(4)}\leq\mu+\sigma) = \left[\Phi(1)\right]^4 \approx (0.8413447)^4 \approx 0.5011.

Thus the probability is just over one half. This is not the probability that a single observation is below the threshold; the fourth power accounts for the requirement that all four observations satisfy it.

Worked Example: Exact Means of the Minimum and Maximum in a Three-Observation Sample

Let $Z_1,Z_2,Z_3$ be IID $N(0,1)$. Find $E[Z_{(3)}]$, the expected maximum, and $E[Z_{(1)}]$, the expected minimum.

The maximum density is $3\phi(z)\Phi(z)^2$, so

E[Z_{(3)}]=3\int_{-\infty}^{\infty}z\phi(z)\Phi(z)^2\,dz.

Since $\phi'(z)=-z\phi(z)$, integration by parts, with vanishing boundary term $\phi(z)\Phi(z)^2$, gives

\int_{-\infty}^{\infty}z\phi(z)\Phi(z)^2\,dz = 2\int_{-\infty}^{\infty}\phi(z)^2\Phi(z)\,dz.

Now $\phi(z)^2=(2\pi)^{-1}e^{-z^2}$. The function $e^{-z^2}$ is even, and $\Phi(z)+\Phi(-z)=1$. Substitution $z\mapsto -z$ therefore shows that $\int_{-\infty}^{\infty}e^{-z^2}\Phi(z)\,dz$ is half of $\int_{-\infty}^{\infty}e^{-z^2}\,dz=\sqrt{\pi}$. Hence

\int_{-\infty}^{\infty}\phi(z)^2\Phi(z)\,dz =\frac{1}{2\pi}\frac{\sqrt{\pi}}{2} =\frac{1}{4\sqrt{\pi}}, \qquad E[Z_{(3)}]=\frac{3}{2\sqrt{\pi}}\approx 0.8463.

The joint distribution of $(-Z_1,-Z_2,-Z_3)$ is the same as that of $(Z_1,Z_2,Z_3)$, and negation converts the maximum into the negative of the minimum. Therefore $E[Z_{(1)}]=-E[Z_{(3)}]=-3/(2\sqrt{\pi})$. For a sample from $N(\mu,\sigma^2)$, the corresponding means are $\mu+3\sigma/(2\sqrt{\pi})$ and $\mu-3\sigma/(2\sqrt{\pi})$.

Central Ranks and Symmetry

A central order statistic has a density concentrated in the middle of the sample, but its exact shape still depends on the rank and sample size. For example, with five observations, the median is the third order statistic. Its density is $30 f(x)F(x)^2[1-F(x)]^2$. In a normal sample this density is symmetric around $\mu$, so its mean is $\mu$. Symmetry also makes the probability that the median falls below $\mu$ equal to one half when the sample size is odd and the distribution is continuous.

Worked Example: The Median of Five Standard Normal Observations

Let $Z_1,\ldots,Z_5$ be IID $N(0,1)$. Write the density and expectation integral for the sample median $Z_{(3)}$, and find its mean and the probability that it is at most zero.

Taking $n=5$ and $k=3$ in the order-statistic density gives

f_{(3)}(z) = \frac{5!}{2!2!}\phi(z)\Phi(z)^2[1-\Phi(z)]^2 = 30\phi(z)\Phi(z)^2[1-\Phi(z)]^2.

Thus the exact expectation integral is

E[Z_{(3)}] = 30\int_{-\infty}^{\infty} z\phi(z)\Phi(z)^2[1-\Phi(z)]^2\,dz.

Under $z\mapsto -z$, the factor $\phi(z)$ is unchanged and $\Phi(-z)=1-\Phi(z)$, so the density is even. The integrand for the mean is odd and integrable; therefore $E[Z_{(3)}]=0$.

For the probability, $Z_{(3)}\leq 0$ means that at least three of the five observations are at most zero. Each observation is at most zero with probability $1/2$, independently. Hence

P(Z_{(3)}\leq 0) = \sum_{j=3}^{5}\binom{5}{j}\left(\frac12\right)^5 = \frac{10+5+1}{32} = \frac12.

The same argument gives $E[X_{(3)}]=\mu$ and $P(X_{(3)}\leq\mu)=1/2$ for five IID $N(\mu,\sigma^2)$ observations.

Using the Formulas Carefully

The density formula gives a systematic route from a normal sample to probabilities and moments: identify the rank $k$, substitute the normal $F$ and $f$, then integrate or use an event description that is simpler. For a probability such as $P(X_{(k)}\leq x)$, an equivalent description is that at least $k$ observations are at most $x$. For minima and maxima, the all-observations event formulas are especially direct.

A common mistake is to treat the maximum of $n$ normal observations as though it were itself $N(\mu,\sigma^2)$, perhaps concluding that its mean is $\mu$. The maximum is not a single original observation: its density contains the factor $nF(x)^{n-1}$, and for a symmetric continuous distribution its expected value is generally shifted upward when $n>1$. In the three-observation calculation above, the shift is exactly $3\sigma/(2\sqrt{\pi})$. By contrast, an odd-sample median from a symmetric normal distribution has mean $\mu$, because its density is symmetric about that point.

Another useful check is standardization. After writing a result for $N(0,1)$, location and scale should enter a mean as $\mu+\sigma$ times the standardized mean, while a variance should acquire a factor $\sigma^2$. A probability at $x$ should depend on $x$ through the standardized threshold $(x-\mu)/\sigma$.

Key takeaway: The $k$th normal order statistic has a density formed from one normal density factor and powers of the normal probabilities on either side. Use that density for exact moment integrals, and use the sample event behind a rank whenever it simplifies a probability calculation.

Check Your Understanding

Use the order-statistic density and normal symmetry to answer these questions.

Write the density of $X_{(2)}$ for an IID sample of size four from $N(\mu,\sigma^2)$.
For an IID sample of size $n$ from $N(\mu,\sigma^2)$, express $P(X_{(1)}>\mu)$ in terms of $n$.
For three IID standard normal observations, what is the expected minimum if the expected maximum is $3/(2\sqrt{\pi})$? Explain the symmetry used.
Write an exact integral for $E[X_{(n)}]$ when the sample is IID $N(\mu,\sigma^2)$.
Explain why the distribution of the maximum is not the same as the distribution of one observation when $n>1$.