Quantile Confidence Intervals From Order Statistics

Why Use Order Statistics for a Quantile?

A simulated sample can estimate a population quantile, such as a median or a high loss percentile. An estimate alone does not show how much Monte Carlo sampling variation it has. An order-statistic confidence interval gives a range for the population quantile using the ranks of the simulated values, without requiring a parametric model for the distribution.

The method is distribution-free in the sense that its coverage calculation does not require knowing the shape or parameters of the underlying continuous distribution. It does require independent observations from the same distribution. In a simulation, that means the outputs used for the interval must be an independent, identically distributed sample from the output distribution of interest.

Definition: For a continuous distribution function $F_X$ and a probability $p$ with $0<p<1$, the population $p$-quantile $q_p$ is the value satisfying $F_X(q_p)=p$. Given an independent sample of size $n$, write its ordered values as $X_{(1)}\le \cdots \le X_{(n)}$, where $X_{(r)}$ is the observation with rank $r$. An interval $[X_{(r)},X_{(s)}]$, for $1\le r<s\le n$, is an order-statistic confidence interval for $q_p$.

The Binomial Coverage Result

The key is to count how many sample observations fall at or below the true quantile. Because $F_X(q_p)=p$, each independent observation has probability $p$ of being at or below $q_p$. Define $B=\#\{j:X_j\le q_p\}$. Under the assumptions above, $B\sim\operatorname{Bin}(n,p)$.

The event that the interval contains $q_p$ can be expressed entirely in terms of $B$. The lower endpoint $X_{(r)}$ is at or below $q_p$ exactly when at least $r$ observations are at or below $q_p$. The upper endpoint $X_{(s)}$ is at or above $q_p$ exactly when no more than $s-1$ observations are below $q_p$. Since the distribution is continuous, an observation equals the fixed value $q_p$ with probability zero. Thus the coverage event is $r\le B\le s-1$.

P\bigl(X_{(r)}\le q_p\le X_{(s)}\bigr) =P(r\le B\le s-1) =\sum_{b=r}^{s-1}\binom{n}{b}p^b(1-p)^{n-b}.

This is the exact confidence level of the interval under the stated assumptions. To construct an interval with confidence at least $1-\alpha$, choose ranks $r$ and $s$ for which the sum is at least $1-\alpha$. Since the binomial distribution is discrete, the achieved confidence will often exceed the requested level rather than equal it.

Specify the quantile and sample size.
Set the target probability $p$, the sample size $n$, and the desired confidence level.

Choose candidate ranks.
For each proposed pair $r<s$, the interval is $[X_{(r)},X_{(s)}]$.

Calculate binomial coverage.
Evaluate $P(r\le B\le s-1)$ for $B\sim\operatorname{Bin}(n,p)$.

Report the endpoints and actual confidence.
Use the selected sample order statistics as endpoints and state the calculated coverage, not just the target coverage.

Choosing Ranks and Allocating Tail Probability

The same coverage calculation can be written in terms of the two excluded tails:

P(r\le B\le s-1) =1-P(B\le r-1)-P(B\ge s).

This form is useful for choosing ranks. A lower-tail exclusion of at most $\alpha_L$ requires $P(B\le r-1)\le\alpha_L$; an upper-tail exclusion of at most $\alpha_U$ requires $P(B\ge s)\le\alpha_U$. If $\alpha_L+\alpha_U\le\alpha$, the resulting interval has coverage at least $1-\alpha$. The allocation need not be equal. For a quantile far from the median, the binomial distribution is asymmetric, so equal allocations can lead to noticeably different rank choices on the two sides.

There is also a useful feasibility check. The widest interval from the observed sample is $[X_{(1)},X_{(n)}]$. It contains $q_p$ unless every observation falls on the same side of the quantile, so its maximum possible coverage is

P\bigl(X_{(1)}\le q_p\le X_{(n)}\bigr) =1-(1-p)^n-p^n.

If this maximum is below the requested confidence level, no choice of order-statistic endpoints can meet the target at that sample size. A wider interval cannot be obtained from the sample order statistics alone.

Worked Example: A 95% Interval for the Median

A hypothetical simulation produces eight independent loss outputs. The sorted values, in dollars, are $14,18,21,23,27,31,36,44$. Construct a distribution-free interval for the population median with at least 95% confidence. Use exact binomial probabilities, not a normal approximation.

Method. The median has $p=0.5$, so the number $B$ of outputs at or below the true median has distribution $\operatorname{Bin}(8,0.5)$. Consider ranks $r=2$ and $s=8$. The interval covers the median when $2\le B\le7$.

There are $2^8=256$ equally likely outcomes for the binomial count. The excluded lower tail is $P(B\le1)=(\binom80+\binom81)/256=(1+8)/256=9/256$. The excluded upper tail is $P(B\ge8)=\binom88/256=1/256$. Therefore,

P(2\le B\le7)=1-\frac{9}{256}-\frac{1}{256} =\frac{246}{256}=0.9609375.

The selected endpoints are $X_{(2)}=\$18$ and $X_{(8)}=\$44$. The interval is $[\$18,\$44]$, with coverage $0.9609$, or 96.09%, rounded to four decimal places for the probability. It exceeds the required 95%. This method avoids assuming a particular loss distribution; the confidence calculation depends on independent sampling and the continuous-distribution condition.

Worked Example: An Interval for the 80th Percentile

A hypothetical model generates 12 independent annual claim totals. Their sorted values, in thousands of dollars, are $31,35,38,41,44,47,50,55,59,64,70,83$. Use the interval $[X_{(8)},X_{(12)}]$ for the population 80th percentile and calculate its exact coverage.

Method. Here $n=12$ and $p=0.8$, so $B\sim\operatorname{Bin}(12,0.8)$. The event for these ranks is $8\le B\le11$. Equivalently, let $K=12-B$ count observations above the quantile. Then $K\sim\operatorname{Bin}(12,0.2)$, and the coverage event is $1\le K\le4$.

Calculate each included probability:

\begin{aligned} P(K=1)&=\binom{12}{1}(0.2)(0.8)^{11}=0.206158430208,\\ P(K=2)&=\binom{12}{2}(0.2)^2(0.8)^{10}=0.283467841536,\\ P(K=3)&=\binom{12}{3}(0.2)^3(0.8)^9=0.236223797248,\\ P(K=4)&=\binom{12}{4}(0.2)^4(0.8)^8=0.132875550720. \end{aligned}

Adding these terms gives $0.206158430208+0.283467841536+0.236223797248+0.132875550720=0.858725619712$. The endpoints are $X_{(8)}=55$ and $X_{(12)}=83$, so the interval is $[55,83]$ thousand dollars, with exact coverage $0.8587$, rounded to four decimal places.

The requested ranks are not symmetric around the sample median because the target quantile is the 80th percentile. A common mistake is to use median-based ranks without checking the binomial probabilities for $p=0.8$. The rank calculation must use the target percentile.

Worked Example: Selecting a 95% Median Interval From 20 Outputs

A hypothetical simulation produces 20 independent values, sorted in thousands of dollars: $9,11,14,17,20,23,26,29,31,34,37,40,43,47,52,58,63,71,80,96$. Determine whether $[X_{(6)},X_{(15)}]$ gives at least 95% confidence for the population median, and state the interval.

Method. For a median, $B\sim\operatorname{Bin}(20,0.5)$. The proposed interval covers the median if $6\le B\le14$. By symmetry of the binomial distribution, the two excluded tails have equal probability. For one tail,

P(B\le5) =\frac{\binom{20}{0}+\binom{20}{1}+\binom{20}{2}+\binom{20}{3}+\binom{20}{4}+\binom{20}{5}}{2^{20}} =\frac{1+20+190+1140+4845+15504}{1048576} =\frac{21700}{1048576}.

Thus the coverage is $1-2(21700/1048576)=0.9586105347$, or $0.9586$ to four decimal places. The endpoints are $X_{(6)}=23$ and $X_{(15)}=52$, giving $[23,52]$ thousand dollars. This interval achieves more than 95% coverage. The calculation avoids rounding a normal approximation to ranks; the exact binomial tails determine the confidence level.

Common Rank Errors and Interpretation

The rank boundaries are easy to shift by one. For $[X_{(r)},X_{(s)}]$, the count must satisfy $r\le B\le s-1$, not $r\le B\le s$. For example, in the first worked example, using $2\le B\le8$ would incorrectly add the $B=8$ outcome. That would give $247/256=0.9648$, rather than the correct $246/256=0.9609$. The upper rank $s$ corresponds to a binomial upper-tail exclusion $B\ge s$.

Do not confuse a confidence interval for the quantile with a range containing a specified fraction of observations. The interval here is intended to cover the fixed population value $q_p$; its coverage comes from repeated sampling.
Do not treat the requested confidence as automatically achieved. Calculate the binomial probability for the chosen ranks and report the actual value.
Check the sample-size limit. If even $[X_{(1)},X_{(n)}]$ has insufficient coverage, no narrower rank interval can meet the target.
State the sampling assumptions. Dependence between simulation outputs or a discrete output distribution can invalidate the exact continuous-iid calculation. For discrete distributions, ties and the quantile convention affect coverage, so this formula should not be applied uncritically.

These intervals quantify sampling uncertainty in a simulated quantile under the assumed output distribution. They do not, by themselves, measure uncertainty in parameters used to specify that distribution. The binomial-rank method is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Check Your Understanding

For each problem, use the binomial count for the stated quantile and interpret the ranks carefully.

For $n=6$ independent continuous observations, consider the median interval $[X_{(1)},X_{(6)}]$. What is its exact coverage? (Answer: $1-2/64=0.9688$, rounded to four decimal places.)
For $n=10$ and the median, calculate the coverage of $[X_{(3)},X_{(8)}]$. (Answer: $1-2(1+10+45)/1024=0.8906$, rounded to four decimal places.)
For $n=8$ and the 75th percentile, calculate the coverage of $[X_{(5)},X_{(8)}]$. (Answer: $P(5\le B\le7)=0.7861$, rounded to four decimal places.)
With $n=12$, can any interval using sample order statistics provide at least 90% confidence for the 95th percentile? Use the widest possible interval to justify your answer. (Answer: No; the maximum coverage is $1-0.05^{12}-0.95^{12}=0.4596$, rounded to four decimal places.)
For $n=8$ and the median, calculate the coverage of $[X_{(2)},X_{(7)}]$. (Answer: $P(2\le B\le6)=1-18/256=0.9297$, rounded to four decimal places.)

What You'll Learn