When Quantile And Distributional Estimates Mislead

When a precise-looking estimate is fragile

A quantile estimate can be numerically precise to several decimal places and still be poorly supported by the data. This is especially likely in a tail, where few observations determine the estimate, or in covariate regions with little comparable information. Distribution-regression curves can also behave implausibly across thresholds, and analyses of censored or missing outcomes depend on assumptions that the observed data cannot fully verify.

The practical task is not to find one diagnostic that certifies an analysis. It is to match each potential weakness to a targeted check, then show whether the substantive conclusion changes under reasonable alternatives. A plot may reveal a problem, but it does not repair it; a sensitivity analysis can show the consequences of an assumption, but cannot prove that the assumption is true.

Tail support and effective sample size

For an upper conditional quantile at level \(\tau\), the relevant tail probability is \(1-\tau\). In a sample of size \(n\), \(n(1-\tau)\) gives a rough count of observations beyond that quantile under a correctly specified target distribution. It is not a guaranteed count in a covariate subgroup, nor does it capture how unevenly observations contribute to an estimator. For example, a 0.95 quantile in \(n=400\) observations corresponds to roughly \(400(0.05)=20\) observations above the target. If the analysis is local to a narrow covariate region, the effective tail support may be much smaller.

Useful checks include plotting the outcome distribution with the estimated quantile marked, reporting counts above and below the estimate, and repeating the display by important covariate groups. For regression estimates, also examine whether the observations most influential for the fitted tail are concentrated in a few covariate regions. Bootstrap estimates that vary substantially across resamples are a warning, particularly when resamples frequently contain very few relevant tail observations. A stable bootstrap distribution is not proof of adequate support: resampling cannot create information absent from the sample.

For nonnegative analytic weights \(w_i\), the Kish effective sample size summarizes weight concentration. It is the size of an equally weighted sample that would have the same contribution of squared normalized weights to the variance of a weighted mean.

To see the rationale, write normalized weights as \(a_i=w_i/\sum_j w_j\), so that \(\sum_i a_i=1\). If independent observations have a common variance \(\sigma^2\), the variance of their weighted mean is \(\sigma^2\sum_i a_i^2\). Equating this to the variance \(\sigma^2/n_{\mathrm{eff}}\) of an equally weighted mean gives \(n_{\mathrm{eff}}=1/\sum_i a_i^2\), which is the displayed formula. This is a diagnostic of concentration, not a universal replacement for a quantile estimator’s sample size or its variance calculation.

Conditional density: a hidden source of quantile instability

Quantile uncertainty depends not only on sample size but also on the density of the outcome near the target quantile. In the simple setting of an independent sample from a continuous distribution with positive density \(f(q_\tau)\) at its \(\tau\)-quantile, the large-sample variance of the sample quantile is approximately \(\tau(1-\tau)/[n f(q_\tau)^2]\). In conditional quantile regression, the corresponding variance also depends on the covariate design and the conditional density at the target quantile. The simple expression below isolates the density effect; it is not a substitute for the regression variance formula.

When the conditional density is small, a modest change in cumulative probability corresponds to a larger change in the outcome value. Thus, even if a standard error calculation is implemented correctly, a low density near the target can make quantile estimates sensitive to sampling variation and to density-estimation choices. A density estimate should be treated cautiously when the bandwidth or local window changes the result substantially.

Crossing and distributional shape checks

A conditional quantile function must be nondecreasing in its probability index: if \(\tau_1<\tau_2\), then the population quantiles satisfy \(Q_{\tau_1}(Y\mid X=x)\leq Q_{\tau_2}(Y\mid X=x)\). Separately estimated quantile regressions can violate this ordering in finite samples. Plot fitted quantiles over a grid of \(\tau\) values at representative covariate values, and also inspect whether crossings occur only in regions with sparse data. The earlier material on diagnosing and repairing quantile crossing covers repair methods; in a diagnostic report, distinguish the raw estimates from any repaired estimates and show whether the repair changes the interpretation.

Distribution regression models the conditional distribution function at outcome thresholds. A valid CDF is nondecreasing in the threshold and takes values between zero and one. Separate fitted binary-response regressions at different thresholds need not automatically satisfy monotonicity. Plot the estimated curves against the outcome threshold for representative covariate values. Downward steps or fitted values outside the unit interval indicate that the fitted collection does not behave like a valid CDF, even if each threshold-specific regression appears individually plausible.

These shape checks are not merely cosmetic. Recovering a quantile from an estimated CDF depends on its ordering across thresholds; nonmonotonicity can make the implied quantile unclear or unstable. Conversely, a smooth-looking curve does not establish that the model is well supported. Check the number of observations near each threshold and compare results under reasonable modeling choices.

Overlap, censoring, and missingness

For distributional comparisons that depend on treatment or selection adjustment, overlap means that the relevant groups have support across the covariate profiles needed for the target comparison. Inspect covariate distributions by group and, when propensity scores are used, plot their distributions by group and locate extreme scores. Report how much of the sample lies in regions with very low estimated treatment probabilities or very low probabilities of remaining observed. Large inverse-probability weights signal that estimates may rely heavily on a few observations. Trimming or restricting the sample can reduce this instability, but it can change the target population; state the resulting target explicitly rather than presenting it as a cost-free numerical fix.

Censoring and missing-outcome adjustments require assumptions about how observation relates to the outcome after conditioning on measured covariates. For conditional independent censoring, the relevant condition is that censoring is independent of the outcome information needed for the target, conditional on the included covariates and at-risk history. For missing outcomes, a common identifying condition is that response is independent of the missing outcome conditional on observed covariates. These assumptions cannot generally be confirmed from observed outcomes alone. Diagnostics can reveal patterns inconsistent with a proposed model, but passing them does not prove conditional independence.

Plot censoring or response rates against observed covariates, compare observed covariate distributions across response patterns, and examine whether modeled probabilities approach zero in important regions. Then conduct sensitivity analyses that vary the assumptions about unobserved outcomes or dependent censoring. Report the direction and magnitude of changes in the target estimate, not only whether a significance threshold is crossed.

A disciplined diagnostic workflow

A useful diagnostic sequence links each concern to evidence and a response. It prevents a common error: treating one favorable plot or one small standard error as proof that all identifying and support conditions are adequate.

Common pitfall: treating diagnostics as a pass/fail test

Wrong reasoning: “The propensity-score plot looks acceptable and the standard error is small, so overlap and the missingness assumptions are satisfied.” This conflates distinct issues. A plot can reveal severe lack of support, but cannot establish that all relevant covariates were measured or that observation is conditionally independent of the unobserved outcome. A small standard error can reflect the variance formula under the model while leaving model misspecification or weak tail support unaddressed.

Correction: Treat plots, density estimates, effective sample sizes, and sensitivity analyses as complementary evidence. Describe what each check can diagnose, what it cannot establish, and how much the estimate changes under plausible alternatives. If conclusions are stable only under a narrow set of assumptions, report that dependence rather than describing the result as robust.