Confidence Intervals for Quantiles and Distribution Functions

Two Different Scales of Uncertainty

A survey may estimate the proportion of a target population whose outcome is at or below a specified threshold, or it may estimate a population percentile such as the median. Both summaries come from the distribution of the outcome, but their uncertainty is expressed on different scales. A confidence interval for a cumulative distribution function (CDF) value is on the probability scale; a confidence interval for a quantile is on the outcome scale, with units such as years, dollars, or minutes.

The distinction matters in practice. A probability-scale interval answers how uncertain the estimated proportion below a fixed threshold is. An outcome-scale interval answers how uncertain the estimated threshold is at which a specified fraction of the population lies below it. The intervals require design-based uncertainty estimates: the variance calculation must reflect the survey design, rather than treating the weighted records as a simple random sample.

The earlier material on weighted empirical CDFs and weighted quantiles establishes how \(\hat F(y)\) and \(\hat Q(p)\) are calculated. Here the focus is how to quantify their uncertainty. Write \(SE\{\hat F(y)\}\) for the design-based standard error of the CDF estimate at a fixed \(y\). It should be obtained using a method appropriate to the survey design, such as linearization or valid replicate weights. The same principle applies to the quantile: its standard error must reflect the design.

Confidence Intervals for CDF Values

For a fixed threshold \(y\), suppose \(\hat F(y)\) is approximately normally distributed around \(F(y)\), with a design-based standard error that estimates its sampling standard deviation. A large-sample two-sided confidence interval is

Here \(1-\alpha\) is the nominal confidence level, and \(z_{1-\alpha/2}\) is the corresponding standard normal critical value. For a 95% interval, \(z_{0.975}\) is approximately 1.96. If the survey variance procedure specifies a \(t\) reference distribution with design degrees of freedom, use its appropriate critical value instead. Because a CDF value must lie between 0 and 1, a simple Wald interval extending outside that range should not be reported as if it were a valid probability interval; use an appropriate bounded procedure in that situation.

The threshold must be treated as fixed for this interpretation. If the threshold is selected after inspecting the same survey estimates, the pointwise interval at that threshold does not automatically provide simultaneous coverage across all possible thresholds. Likewise, a collection of pointwise CDF intervals is not, without an additional procedure, a simultaneous confidence band for the whole CDF.

Confidence Intervals for Quantiles

A quantile interval is more demanding because it translates uncertainty in probabilities into uncertainty in outcome values. When the population distribution is continuous near the target quantile and has a positive, reasonably smooth density there, a local linear approximation gives a useful standard error. Let \(q=Q(p)\), and write \(f(q)\) for the population density at \(q\). The design-based variance of \(\hat F(q)\) measures uncertainty in the estimated CDF at the true quantile. A plug-in approximation to the quantile standard error is

where \(\hat f(\hat Q(p))\) estimates the density near the estimated quantile. The numerator is measured on the probability scale and the density has units of probability per outcome unit. Their ratio therefore has outcome units, as a quantile standard error must.

Under these conditions, a large-sample interval is \(\hat Q(p)\pm z_{1-\alpha/2}SE\{\hat Q(p)\}\). This is an approximation, not a universal formula for every outcome distribution. In particular, ties and discrete outcomes can make the relationship between CDF uncertainty and quantile uncertainty non-smooth. The density-based calculation should not be applied mechanically in those settings.

Why the Quantile Standard Error Has This Form

The approximation follows by expanding the CDF near the true quantile. For a continuous distribution with positive density at \(q\), \(F(q)=p\), and for a small change \(h\),

The estimated quantile is approximately the value at which the estimated CDF reaches \(p\). Replacing the population CDF by the estimated CDF and applying the local expansion gives

Since \(F(q)=p\), rearrangement yields

Multiplying a random error by a constant divides its variance by the square of that constant. Therefore, to first order,

In practice, \(q\), the CDF standard error at \(q\), and the density are unknown, so they are replaced by estimates. The approximation explains why quantiles are especially uncertain where the distribution is relatively flat: a small change in probability then corresponds to a larger change in the outcome value.

Quantile Intervals by Inverting CDF Intervals

A second way to express quantile uncertainty is to invert CDF intervals. At each fixed outcome value \(y\), form a lower and upper confidence limit \(L(y)\) and \(U(y)\) for \(F(y)\). For a continuous distribution at the target quantile, \(F(Q(p))=p\). The quantile interval can then be obtained from the outcome values where the CDF limits cross \(p\): the lower endpoint is where the upper CDF limit reaches \(p\), and the upper endpoint is where the lower CDF limit reaches \(p\).

More precisely, when the limits are ordered monotonically in \(y\), the lower endpoint is the largest \(y\) for which \(U(y)<p\), and the upper endpoint is the smallest \(y\) for which \(L(y)\ge p\). The logic is that an upper limit below \(p\) supports that the quantile lies above \(y\), while a lower limit at or above \(p\) supports that it lies at or below \(y\). In continuous settings, inversion of appropriate pointwise intervals can give a confidence set for the quantile. With only a finite grid of outcome values, the crossing locations are approximate and may need interpolation.

The inversion argument relies on the behavior of the CDF at the target quantile and on appropriately constructed limits. Pointwise CDF intervals do not constitute a simultaneous confidence band over all \(y\). If the distribution is discrete and \(F(Q(p))\) can exceed \(p\) because of a jump, the continuous-crossing argument does not directly give the same coverage guarantee. Discrete outcomes and ties require methods that account for that structure rather than an unexamined density approximation.

Common Mistake and Practical Checks

A common mistake is to confuse the units or targets of the two intervals. For example, an analyst might report a CDF interval of 0.55 to 0.69 as an interval for the median, or report an interval of 17 to 20 minutes as uncertainty in the proportion below a threshold. Neither is correct: the first describes a probability at a specified outcome value, and the second describes an outcome quantile.

A second pitfall is to treat weighted observations as independent, equally informative records when calculating uncertainty. The weighted estimate alone does not determine its standard error. The variance procedure must use the available design information, including the relevant stratification, clustering, and weights, or an appropriate replicate-weight method. A final check is to verify units: a CDF estimate and its standard error are dimensionless, while a quantile and its standard error carry the units of the measured outcome.