Sampling Uncertainty in Empirical Quantiles

Why a Sample Quantile Is Uncertain

A sample quantile is determined by the rank of an observation, not by averaging all observations. For example, a sample 90th percentile is near the point above which about 10% of observations lie. But in a finite sample, the number of observations in that upper tail varies. Consequently, the value at the estimated percentile can shift substantially from sample to sample, particularly when only a few observations are expected beyond it.

This tutorial develops two ways to assess that uncertainty. Order-statistic intervals use only the quantile level and the sample size, and can provide exact coverage under a continuous-distribution assumption. A large-sample standard error gives a second perspective when the density near the quantile is known or can be estimated. Both methods make the limited information in a small tail visible.

Definition: Let $X_1,\ldots,X_n$ be independent observations from a common distribution with cumulative distribution function $F_X$. The population $p$-quantile is $q_p=\inf\{x:F_X(x)\ge p\}$, where $0<p<1$. Write the ordered observations as $X_{(1)}\le \cdots\le X_{(n)}$. A common empirical-quantile convention is $\hat q_p=X_{(\lceil np\rceil)}$, with the rank restricted to $1,\ldots,n$. Other software conventions may interpolate between adjacent order statistics, so the convention should be stated when reporting a result.

The value $\hat q_p$ is a point estimate, not a guarantee that the population quantile is close to it. Its uncertainty concerns how the sample rank would vary across repeated independent samples. The methods below quantify that variation rather than treating the observed order statistic as fixed truth.

Exact Intervals from Order Statistics

Assume the observations are independent and identically distributed from a continuous distribution, with population quantile $q_p$. Define $K$ as the number of sample observations at or below $q_p$. Continuity means the probability of an observation exactly equal to $q_p$ is zero, so $K\sim\operatorname{Bin}(n,p)$. This converts uncertainty about the quantile into a binomial-count calculation.

Theorem: Under the independent, identically distributed, continuous model, the interval $[X_{(r)},X_{(s)}]$, for ranks $1\le r<s\le n$, contains $q_p$ with probability $$ P\bigl(X_{(r)}\le q_p\le X_{(s)}\bigr) =P(r\le K\le s-1) =\sum_{k=r}^{s-1}\binom{n}{k}p^k(1-p)^{n-k}, \qquad K\sim\operatorname{Bin}(n,p). $$

The rank conditions explain the endpoints. The lower endpoint $X_{(r)}$ is at or below $q_p$ precisely when at least $r$ observations are at or below $q_p$, or $K\ge r$. The upper endpoint $X_{(s)}$ is at or above $q_p$ precisely when at most $s-1$ observations are at or below $q_p$, or $K\le s-1$. Both conditions hold exactly when $r\le K\le s-1$. This is the reason the binomial sum gives the interval’s coverage.

For a desired confidence level, choose ranks whose binomial probability is at least that level and, where possible, close to it. Exact coverage is discrete: there may be no pair of ranks giving exactly 90% or 95%. The interval may therefore be conservative, meaning its actual coverage exceeds the target. A wider choice of ranks generally increases coverage, but also makes the interval less precise.

Set the target.
Specify the quantile level $p$, sample size $n$, and desired coverage.

Translate the ranks.
For proposed endpoints $X_{(r)}$ and $X_{(s)}$, calculate $P(r\le K\le s-1)$ for $K\sim\operatorname{Bin}(n,p)$.

Use the observed data.
Sort the sample and report the values at ranks $r$ and $s$, together with the calculated coverage and assumptions.

Worked Example: A Median Interval from Eight Observations

A hypothetical insurer reviews eight independent claim amounts from the same continuous severity distribution. In thousands of dollars, the sorted observations are $2,3,4,5,6,7,9,12$. Construct an order-statistic interval for the population median using ranks 2 and 7, calculate its coverage, and check the endpoints. The method is an exact binomial count; a common mistake is to treat the interval as having a predetermined 95% coverage without checking it.

Step 1: Identify the quantile and count distribution. For the median, $p=0.5$, and $n=8$. Thus $K\sim\operatorname{Bin}(8,0.5)$. The proposed interval is $[X_{(2)},X_{(7)}]$, so its coverage is $P(2\le K\le6)$.

Step 2: Calculate the coverage. For $k=0,\ldots,8$, the binomial coefficients are $1,8,28,56,70,56,28,8,1$, each divided by $2^8=256$. The excluded counts are $K=0,1,7,8$, with total probability $(1+8+8+1)/256=18/256$. Therefore,

P(2\le K\le6)=1-\frac{18}{256}=\frac{238}{256}=0.9296875.

The interval has 92.97% coverage, rounded to two decimals in percent. It is not a 95% interval: the chosen ranks do not deliver that coverage.

Step 3: Read the endpoints and check. From the sorted data, $X_{(2)}=3$ and $X_{(7)}=9$, so the interval is $[3,9]$ thousand dollars. The observed sample median under the stated convention is $X_{(\lceil8(0.5)\rceil)}=X_{(4)}=5$ thousand dollars, which lies inside the interval. This check confirms the ranks were applied correctly; it does not change the calculated repeated-sampling coverage.

Tail Quantiles and the Number of Exceedances

For an upper-tail quantile, the count above the quantile is often the most direct way to see how much information the sample contains. If the quantile level is $p$, the expected number of observations above $q_p$ is $n(1-p)$. When this expected count is small, a few observations moving in or out of the tail can move the estimated quantile markedly.

Worked Example: How Often a 95th Percentile Is Above the Sample Maximum

Suppose 20 independent claim amounts come from a continuous distribution, and $q_{0.95}$ is its population 95th percentile. Calculate the probability that all 20 observations fall below $q_{0.95}$, and interpret what this says about estimating the tail. Use the binomial count; the trap is to assume that a 95th percentile must be exceeded in every sample.

For each observation, the probability of being at or below $q_{0.95}$ is $0.95$. By independence, the probability all 20 observations are at or below it is

P(K=20)=0.95^{20}=0.3584859\ldots

To four decimal places, this is $0.3585$. Thus, in about 35.85% of samples, even the sample maximum is below the population 95th percentile. Equivalently, the expected number of observations above that percentile is $20(0.05)=1$. A sample of this size has little direct information about the far tail, despite having 20 observations in total.

The same calculation applies to other tail levels: the probability of observing no values above $q_p$ is $p^n$, and the expected number above it is $n(1-p)$. These are not confidence intervals, but they help explain why tail-quantile estimates can be unstable.

A Large-Sample Standard Error

A second approximation connects quantile uncertainty to the density near the quantile. Assume independent observations from a continuous distribution with a density $f_X$ that is positive and sufficiently smooth near $q_p$. For a large sample, the approximate standard error of the empirical $p$-quantile is

\operatorname{SE}(\hat q_p)\approx \frac{\sqrt{p(1-p)/n}}{f_X(q_p)}.

The numerator reflects the sampling variation in the proportion of observations below the quantile. At the true quantile, the empirical proportion below it has approximate standard deviation $\sqrt{p(1-p)/n}$. Near $q_p$, a small change $\Delta x$ in the threshold changes the population proportion by approximately $f_X(q_p)\Delta x$, because $F_X(q_p+\Delta x)-F_X(q_p)\approx f_X(q_p)\Delta x$. Dividing the proportion’s standard deviation by this local rate of change gives the stated standard error.

A small density means that a sizable change in the claim amount corresponds to only a small change in cumulative probability. That makes the quantile more variable. The approximation is not a substitute for the exact order-statistic calculation in a small sample; it also depends on a credible value or estimate of the density at the quantile. If the density is irregular or close to zero there, the approximation can be unreliable.

Worked Example: Approximate Uncertainty in a High Quantile

A hypothetical sample has $n=400$ independent claim amounts. Its empirical 95th percentile is $\hat q_{0.95}=\$1{,}200$. For illustration, assume a smooth continuous severity distribution with density near the population 95th percentile estimated as $f_X(q_{0.95})=0.0005$ per dollar. Use the large-sample standard error to form an approximate 95% interval. The method is a normal approximation; the important trap is to omit the density from the denominator or to treat a density as dimensionless.

Step 1: Calculate the probability-scale standard deviation.

\sqrt{\frac{p(1-p)}{n}} =\sqrt{\frac{0.95(0.05)}{400}} =\sqrt{0.00011875} \approx 0.0108972.

Step 2: Convert to dollar units using the density. The density is per dollar, so dividing by it gives dollars:

\operatorname{SE}(\hat q_{0.95}) \approx\frac{0.0108972}{0.0005} =\$21.7944.

Step 3: Form the approximate interval. Using the standard normal multiplier $1.96$, the margin is $1.96(\$21.7944)=\$42.7170$. The interval is

\$1{,}200\pm \$42.7170 \quad\Longrightarrow\quad [\$1{,}157.28,\ \$1{,}242.72].

The endpoints are rounded to the nearest cent. This is an approximate interval under the stated smooth-density and large-sample assumptions, not an exact distribution-free interval. Its width depends directly on the density estimate; an inaccurate density can make the reported uncertainty misleading.

Choosing and Reporting an Uncertainty Measure

An order-statistic interval is useful when a distribution-free statement is desired and the independent, continuous sampling assumptions are appropriate. It reports actual sample values as endpoints and obtains its coverage from a binomial calculation. The large-sample standard error is useful for summarizing local precision, but it requires a density at the target quantile and an approximation that may be poor in small samples.

For a high percentile, report the sample size as well as the estimate and uncertainty measure. A 99th percentile based on 100 observations has an expected number of exceedances of only $100(0.01)=1$. A seemingly precise dollar estimate should therefore be interpreted in light of how few observations can determine its rank.

The continuous-distribution assumption matters. With a discrete distribution or tied observations, the count at or below $q_p$ need not have success probability exactly $p$, and the binomial coverage formula above does not apply as written. In that setting, state the quantile convention and use a method appropriate to the discrete data rather than treating ties as if the continuous exact calculation still held.

Wrong: Call $[X_{(2)},X_{(7)}]$ a 95% interval because it looks broad around the median.
Correction: Calculate $P(2\le K\le6)$ for $K\sim\operatorname{Bin}(8,0.5)$; it is $0.9297$, not $0.95$.
Wrong: Treat a large total sample as proof that an extreme quantile is precise.
Correction: Check the expected tail count $n(1-p)$ and the relevant order-statistic coverage. A high percentile may depend on very few observations.

Key takeaway: Quantile uncertainty is rank uncertainty. Under independent continuous sampling, binomial probabilities give the exact coverage of intervals made from order statistics. High tail levels leave few observations beyond the target, so tail estimates can remain imprecise even when a sample appears substantial.

Check Your Understanding

Use the stated order-statistic convention and assumptions. Round probabilities to four decimal places where needed.

Six independent observations come from a continuous distribution. For the population median, consider $[X_{(1)},X_{(6)}]$. Express its coverage as a binomial probability and calculate it. (Answer: $P(1\le K\le5)$, $K\sim\operatorname{Bin}(6,0.5)$; coverage $=0.9688$.)
For $n=10$ observations from a continuous distribution, an interval for the median is $[X_{(2)},X_{(9)}]$. Calculate its exact coverage. (Answer: $P(2\le K\le8)$, $K\sim\operatorname{Bin}(10,0.5)$; coverage $=1002/1024=0.9785$.)
A sample contains 12 independent observations, and the target is $q_{0.90}$. Find the expected number above the target and the probability that none are above it. (Answer: Expected number $=1.2$; probability $=0.90^{12}=0.2824$.)
For $n=225$, $p=0.80$, and density $f_X(q_p)=0.004$ per unit, calculate the large-sample standard error of the empirical quantile. (Answer: $\sqrt{0.80(0.20)/225}/0.004\approx 6.6667$ units.)
For $n=8$ and $p=0.75$, use $[X_{(3)},X_{(8)}]$. Calculate its exact coverage from the binomial count. (Answer: $P(3\le K\le7)$, $K\sim\operatorname{Bin}(8,0.75)$; coverage $=58698/65536\approx0.8957$.)

What You'll Learn