What Does It Mean to Be Uncertain About a Conditional Law?
A fitted conditional distribution is an estimate, not a known object. For a covariate value \(x\), write \(F_x(y)=P(Y\leq y\mid X=x)\) for the conditional CDF, as in the earlier material on conditional CDFs and quantile functions. A confidence procedure should describe uncertainty about this whole function or law, rather than report an estimate as if it were exact.
There are several distinct coverage questions. One interval may cover \(F_{x_0}(y_0)\) at one specified outcome \(y_0\). A band may cover \(F_{x_0}(y)\) simultaneously for every \(y\). A confidence set may contain the entire conditional law \(F_{x_0}\). These are not interchangeable claims. In particular, pointwise intervals at many values of \(y\) do not automatically make a simultaneous band.
The probability in these definitions refers to repeated samples. When \(X\) is continuous, a conditional law at an individual \(x_0\) requires a specified version of the conditional distribution; estimation there also relies on \(x_0\) being in a region supported by the design and on suitable regularity. Support constraints and extrapolation are not repaired by attaching a confidence band.
Pointwise Intervals and Simultaneous Bands
A pointwise interval targets one number, such as \(F_{x_0}(y_0)\). For example, if the conditional observations are independent and identically distributed within a discrete covariate group, the indicators \(I(Y_i\leq y_0)\) are Bernoulli with success probability \(F_{x_0}(y_0)\). A normal approximation for their sample proportion gives a convenient pointwise interval when the group sample size is adequate and the probability is not too close to zero or one.
Worked Example: A Pointwise CDF Interval
Suppose a group with \(X=x_0\) contains \(n=100\) independent observations, of which 64 satisfy \(Y_i\leq 2\). The target is the single value \(F_{x_0}(2)\). The empirical estimate is \(\hat F_{x_0}(2)=64/100=0.64\).
Using the estimated Bernoulli standard error gives \( \widehat{\operatorname{se}}=\sqrt{0.64(1-0.64)/100}=\sqrt{0.002304}=0.048. \) A 95% normal-approximation interval is therefore \(0.64\pm 1.96(0.048)=0.64\pm0.09408\), or approximately \([0.546,0.734]\).
This is a pointwise interval: its nominal coverage concerns \(F_{x_0}(2)\) alone. Repeating the calculation at many thresholds does not by itself give 95% coverage of the entire CDF.
For simultaneous coverage, calibrate a statistic that measures the largest error over the outcome domain. A common choice is the supremum norm. Let \(\hat F_{x_0}\) estimate \(F_{x_0}\), and suppose a bootstrap procedure produces a critical value \(c^*_{1-\alpha}\) for the magnitude of the estimation error on the appropriate scale. A band has the form \(\hat F_{x_0}(y)\pm c^*_{1-\alpha}\hat s_n(y)\), clipped to \([0,1]\) when necessary, where \(\hat s_n(y)\) is an estimated standard-error scale. The critical value must be calibrated for the supremum over \(y\), not separately at each \(y\).
To justify the claim, define \(M_n=\sup_{y\in T}|Z_n(y)|\). The event that the un-clipped band contains the target at every \(y\) is exactly \(\{M_n\leq c^*_{1-\alpha}\}\). By the assumed convergence in distribution, \(M_n\xrightarrow{d}Z\); by consistency, \(c^*_{1-\alpha}\xrightarrow{p}c_{1-\alpha}\). Continuity of the distribution of \(Z\) at \(c_{1-\alpha}\) then gives \(P(M_n\leq c^*_{1-\alpha})\to P(Z\leq c_{1-\alpha})=1-\alpha\). Clipping endpoints to \([0,1]\) cannot remove the true CDF value, which itself lies in \([0,1]\). This proves the result. \(\square\)
The theorem states the calibration conditions rather than asserting that every bootstrap works. The process approximation, standard-error estimates, and bootstrap consistency must be justified for the estimator and sampling design in use. For example, a bootstrap that fails to reproduce the estimator’s bias or dependence structure need not provide a valid critical value.
Worked Example: A Simultaneous Band from DKW
Suppose \(m=25\) observations in a discrete covariate group are independent draws from \(F_{x_0}\). Let \(\hat F_{x_0}\) be their empirical CDF. The Dvoretzky–Kiefer–Wolfowitz inequality states that, for every distribution \(F_{x_0}\) and every \(\epsilon>0\), \(P\{\sup_y|\hat F_{x_0}(y)-F_{x_0}(y)|>\epsilon\}\leq 2e^{-2m\epsilon^2}\).
Set the upper bound equal to \(0.05\). Solving \(2e^{-2m\epsilon^2}=0.05\) gives \(\epsilon=\sqrt{\log(2/0.05)/(2m)}=\sqrt{\log(40)/50}\approx\sqrt{0.07378}\approx0.272\). Thus the band \([\max(0,\hat F_{x_0}(y)-0.272),\min(1,\hat F_{x_0}(y)+0.272)]\) has simultaneous coverage at least 95%.
If \(\hat F_{x_0}(0.5)=17/25=0.68\), the band at \(y=0.5\) is approximately \([0.408,0.952]\). Its guarantee, however, is stronger than that one interval: the same event covers every threshold \(y\) at once.
Inverting Distributional Tests
A general route to confidence sets is to test candidate conditional laws and retain those that are not rejected. Suppose for each candidate law \(G\) there is a test of \(H_0:F_{x_0}=G\) whose rejection probability under that null is at most \(\alpha\). Define \(C_n(x_0)\) as the set of candidates not rejected.
Under the true law \(F_{x_0}\), the event \(F_{x_0}\notin C_n(x_0)\) means exactly that the test of the true null \(H_0:F_{x_0}=F_{x_0}\) rejects. Its probability is at most \(\alpha\) by the size assumption. Taking complements yields \(P\{F_{x_0}\in C_n(x_0)\}\geq1-\alpha\). This argument proves coverage directly; it does not require the tests for false candidate laws to have high power. \(\square\)
One possible test statistic compares a candidate CDF with an estimated CDF using a distributional distance, such as \(\sup_y|\hat F_{x_0}(y)-G(y)|\). A bootstrap or other calibration can determine which distances are compatible with the data. Valid coverage still depends on calibration under the true null; merely choosing a plausible-looking distance threshold is not enough.
Worked Example: Translating a Bootstrap Test into a Set
Suppose observations at \(x_0\) take values in \(\{0,1,2\}\), and the observed counts are \(4,4,2\), respectively, out of 10. The empirical CDF at the two nontrivial thresholds is \(\hat F_{x_0}(0)=0.4\) and \(\hat F_{x_0}(1)=0.8\). Assume a bootstrap test has calibrated the supremum-distance cutoff as \(0.15\).
A candidate law \(G\) on these three values is described by probabilities \(q_0,q_1,q_2\geq0\) with \(q_0+q_1+q_2=1\). Its CDF values are \(G(0)=q_0\) and \(G(1)=q_0+q_1\). Retaining candidates whose largest CDF discrepancy is at most \(0.15\) gives \(|q_0-0.4|\leq0.15\) and \(|q_0+q_1-0.8|\leq0.15\).
Equivalently, the confidence set contains the probability vectors satisfying \(0.25\leq q_0\leq0.55\), \(0.65\leq q_0+q_1\leq0.95\), and \(q_0+q_1+q_2=1\), with all probabilities nonnegative. For instance, \((0.4,0.4,0.2)\) is included because its two CDF discrepancies are zero. The vector \((0.6,0.2,0.2)\) is excluded because its discrepancy at zero is \(|0.6-0.4|=0.2>0.15\). This construction describes uncertainty about a whole law, subject to the stated test calibration.
Coverage Depends on the Probability Metric
A simultaneous CDF band corresponds to coverage in the Kolmogorov distance \(d_K(F,G)=\sup_y|F(y)-G(y)|\). It does not generally imply coverage in total variation, Wasserstein distance, or another probability metric. These metrics measure different features of distributions, so a set calibrated for one cannot automatically be interpreted as a confidence set for another.
Worked Example: Close CDFs but Different Total Variation
For an integer \(m\geq1\), let \(P\) put mass \(1/(2m)\) at each point \(i/(2m)\), for \(i=1,\ldots,2m\). Let \(Q\) put mass \(1/m\) at the even-indexed points \(2j/(2m)\), for \(j=1,\ldots,m\), and zero mass at the odd-indexed points.
After each even-indexed point, the two CDFs agree. Between an odd-indexed point and the next even-indexed point, \(F_P\) exceeds \(F_Q\) by \(1/(2m)\). Hence \(d_K(P,Q)=1/(2m)\), which can be made arbitrarily small by increasing \(m\).
For discrete distributions on the same support, total variation is half the sum of absolute differences in their point masses. At each of the \(2m\) points, the absolute difference is \(1/(2m)\): odd points contribute \(1/(2m)\), and even points contribute \(|1/(2m)-1/m|=1/(2m)\). Therefore \(d_{\mathrm{TV}}(P,Q)=\tfrac12(2m)(1/(2m))=1/2\). Small CDF discrepancy alone has not controlled total variation.
Common Pitfall
- Wrong reasoning: “I computed a 95% interval at each of 100 thresholds, so the entire CDF is covered with probability 95%.” Each interval has its own pointwise guarantee; the probability that all 100 cover can be lower.
- Correction: For a simultaneous claim, calibrate a statistic that accounts for the joint error, such as the supremum over thresholds, or use a valid test-inversion procedure. State which metric or functionals the resulting set covers.
- Another overreach: A band in the supremum CDF distance is treated as a band in total variation or Wasserstein distance. The example above shows why this does not follow without additional assumptions or a separate metric-specific bound.
Check Your Understanding
For each question, identify the coverage claim and the reasoning needed to support it.
- What is the difference between coverage at a fixed threshold and simultaneous coverage over all thresholds?
- In the DKW example, why does the computed band provide a stronger statement than an interval at \(y=0.5\)?
- State the test-size condition that makes inversion yield a confidence set with coverage at least \(1-\alpha\).
- Why does a small Kolmogorov distance not, by itself, guarantee small total variation distance?
- What additional calibration issue must be addressed before interpreting a bootstrap critical value as providing valid coverage?