Weak Convergence of Conditional Distribution Estimates

What Does Convergence of a Conditional Distribution Mean?

A conditional distribution estimate is a probability law indexed by a covariate value. Its convergence can be assessed at one specified value $x$, or after averaging its error over covariates drawn from a target population. Those are different guarantees: an estimator can be accurate for almost all covariates while remaining inaccurate at a particular point.

Let $\mathcal X$ be a covariate space and $\mathcal Y$ a Polish response space, meaning a complete separable metric space. Write $P_x$ for the target conditional law of the response given covariate $x$, and $\widehat P_n(\cdot\mid x)$ for its estimate. For random estimates, $\widehat P_n$ is itself random. We assume these are measurable probability kernels: for each measurable response set $A$, the map $x\mapsto P_x(A)$, and likewise $x\mapsto\widehat P_n(A\mid x)$, is measurable.

Definition: Probability measures $Q_n$ on $\mathcal Y$ converge weakly to $Q$, written $Q_n\Rightarrow Q$, if $$ \int g(y)\,Q_n(dy)\longrightarrow \int g(y)\,Q(dy) $$ for every bounded continuous function $g:\mathcal Y\to\mathbb R$. For a fixed covariate value $x$, weak convergence of the conditional estimates means $\widehat P_n(\cdot\mid x)\Rightarrow P_x$. If the estimates are random, convergence in probability in this sense means that each such test-function difference converges to zero in probability.

It is often useful to express weak convergence through one metric. Let $\rho$ be a metric generating the topology of $\mathcal Y$, and set $\rho_b(y,z)=\min\{\rho(y,z),1\}$. The bounded-Lipschitz distance is

d_{\mathrm{BL}}(Q_1,Q_2) = \sup_{\substack{\lVert g\rVert_\infty\leq 1\\ |g(y)-g(z)|\leq \rho_b(y,z)}} \left|\int g\,dQ_1-\int g\,dQ_2\right|.

The bounded-Lipschitz metrization theorem states that, on a Polish space, $d_{\mathrm{BL}}(Q_n,Q)\to0$ if and only if $Q_n\Rightarrow Q$. Thus, for a fixed $x$, weak convergence is equivalent to $d_{\mathrm{BL}}(\widehat P_n(\cdot\mid x),P_x)\to0$. For random estimates, convergence of this distance to zero in probability is equivalent to convergence in probability against every bounded continuous test function. The metric is bounded by $2$, a fact that will matter when averaging over covariates.

Pointwise and Integrated Guarantees

A pointwise guarantee fixes $x$ first and asks whether the estimated conditional law approaches the target law there. An integrated guarantee instead specifies a probability distribution $Q$ on $\mathcal X$ and measures error for covariates drawn from $Q$. The distribution $Q$ must be stated: it determines which parts of the covariate space count toward the guarantee.

Definition: The integrated bounded-Lipschitz error under $Q$ is $$ D_n(Q)=\int_{\mathcal X} d_{\mathrm{BL}}\!\left(\widehat P_n(\cdot\mid x),P_x\right)\,Q(dx). $$ For random estimates, $D_n(Q)$ is random. Integrated convergence means $D_n(Q)\to0$ for deterministic estimates, or $D_n(Q)\xrightarrow{p}0$ for random estimates.

Because the integrand is bounded, integrated convergence has a useful equivalent interpretation: the conditional-law error tends to zero in $Q$-measure. Indeed, for any $\varepsilon>0$,

Q\!\left\{x: d_{\mathrm{BL}}(\widehat P_n(\cdot\mid x),P_x)>\varepsilon\right\} \leq \frac{D_n(Q)}{\varepsilon}.

Conversely, if these probabilities tend to zero for every $\varepsilon>0$, then for any $\varepsilon>0$, using $d_{\mathrm{BL}}\leq2$,

D_n(Q) \leq \varepsilon+ 2Q\!\left\{x: d_{\mathrm{BL}}(\widehat P_n(\cdot\mid x),P_x)>\varepsilon\right\}.

Taking a limit superior and then letting $\varepsilon$ decrease to zero gives $D_n(Q)\to0$. The same reasoning applies to random estimates in probability, with the measure of the error set treated as a random quantity.

Worked Example: Normal Conditional Laws at a Fixed Covariate

Suppose the target law at a fixed $x$ is $P_x=N(\mu(x),\sigma^2(x))$, where $\sigma(x)>0$, and the estimate is $\widehat P_n(\cdot\mid x)=N(\mu_n(x),\sigma_n^2(x))$, where $\sigma_n(x)>0$. Assume $\mu_n(x)\to\mu(x)$ and $\sigma_n(x)\to\sigma(x)$.

Let $Z\sim N(0,1)$ and couple the two laws using $Y_n=\mu_n(x)+\sigma_n(x)Z$ and $Y=\mu(x)+\sigma(x)Z$. Then $Y_n\sim\widehat P_n(\cdot\mid x)$, $Y\sim P_x$, and $Y_n\to Y$ almost surely. For any bounded continuous $g$, continuity gives $g(Y_n)\to g(Y)$ almost surely. Since $|g(Y_n)|\leq\lVert g\rVert_\infty$, the dominated convergence theorem yields

\int g\,d\widehat P_n(\cdot\mid x) =E[g(Y_n)] \longrightarrow E[g(Y)] =\int g\,dP_x.

Thus the estimated conditional laws converge weakly at this $x$. The argument establishes pointwise convergence; it does not by itself say anything about how errors behave over all covariate values.

From Integrated Metric Error to Test-Function Error

Integrated convergence of $d_{\mathrm{BL}}$ gives a strong, reusable conclusion: for each bounded continuous response test function, the absolute conditional expectation error also vanishes on average over $Q$. This conclusion does not require the test function itself to be Lipschitz.

Theorem: Suppose $D_n(Q)\to0$ for deterministic measurable probability kernels. Then, for every bounded continuous $g:\mathcal Y\to\mathbb R$, $$ \int_{\mathcal X}\left| \int g\,d\widehat P_n(\cdot\mid x)-\int g\,dP_x \right|Q(dx)\longrightarrow0. $$ If the estimates are random and $D_n(Q)\xrightarrow{p}0$, the displayed integral converges to zero in probability.

Proof. First consider deterministic kernels. The convergence $D_n(Q)\to0$ implies convergence of the conditional-law distance to zero in $Q$-measure, by the inequality above. From any subsequence, one can select a further subsequence along which the distances converge to zero for $Q$-almost every $x$. For those $x$, bounded-Lipschitz metrization gives weak convergence, so the test-function differences converge to zero. Their absolute values are bounded by $2\lVert g\rVert_\infty$. The dominated convergence theorem therefore makes their $Q$-integrals tend to zero along this further subsequence. Since every subsequence has a further subsequence with this limit, the full sequence of integrals tends to zero.

For random estimates, take any subsequence. Convergence in probability of $D_n(Q)$ allows a further subsequence along which $D_n(Q)\to0$ almost surely. For almost every realization, the deterministic argument just given shows that the integrated test-function error tends to zero. The subsequence characterization of convergence in probability then gives the claimed convergence in probability for the original sequence. $\square$

There is also a useful converse under a stronger formulation. If the integrated bounded-Lipschitz discrepancy tends to zero, that is precisely integrated weak convergence in this metric. Merely checking one or a few test functions, however, is not enough to establish weak convergence: those functions may fail to detect changes in other aspects of the response distribution.

Worked Example: A Small Error Almost Everywhere but Failure at One Point

Let $\mathcal X=[0,1]$, let $Q$ be uniform, and set $P_x=N(0,1)$ for every $x$. Define

\widehat P_n(\cdot\mid x)= \begin{cases} N(1,1),&0\leq x\leq 1/n,\\ N(0,1),&1/n<x\leq1. \end{cases}

The estimate is incorrect on an interval of $Q$-probability $1/n$, so, since $d_{\mathrm{BL}}\leq2$,

D_n(Q)\leq 2Q([0,1/n])=\frac{2}{n}\longrightarrow0.

But at $x=0$, the estimate is always $N(1,1)$, which does not converge weakly to $N(0,1)$. For example, $g(y)=\min\{1,\max\{-1,y\}\}$ is bounded and continuous, and its expectation under $N(1,1)$ is strictly greater than under $N(0,1)$: $g$ is nondecreasing and the shifted normal has a strictly larger value with positive probability. Thus integrated convergence does not guarantee convergence at every specified $x$. The point $x=0$ has zero $Q$-probability, so its persistent error does not prevent the integrated error from vanishing.

When Pointwise Convergence Does Imply Integrated Convergence

For deterministic estimates, pointwise weak convergence for $Q$-almost every $x$ is sufficient for integrated convergence. In fact, the bounded-Lipschitz distance then converges to zero for $Q$-almost every $x$, and it is bounded by $2$. Dominated convergence gives

\int d_{\mathrm{BL}}\!\left(\widehat P_n(\cdot\mid x),P_x\right)Q(dx) \longrightarrow0.

For random estimates, suppose that for $Q$-almost every fixed $x$, the distance $d_{\mathrm{BL}}(\widehat P_n(\cdot\mid x),P_x)$ converges to zero in probability. Boundedness implies its expectation tends to zero at each such $x$. Applying dominated convergence to these expectations gives $E[D_n(Q)]\to0$; Markov's inequality then yields $D_n(Q)\xrightarrow{p}0$. This is an almost-everywhere-in-$x$ condition. Pointwise convergence at a single $x$, or only on a set with small $Q$-probability, cannot give the same population-wide guarantee.

Worked Example: Pointwise Normal Convergence with an Integrated Bound

Let $X\sim Q$, where $Q$ is uniform on $[0,1]$, and suppose $P_x=N(0,1)$ while $\widehat P_n(\cdot\mid x)=N(x/n,1)$. Couple the two laws with $Z\sim N(0,1)$. A test function in the definition of $d_{\mathrm{BL}}$ is bounded by $1$ and is $1$-Lipschitz, so

d_{\mathrm{BL}}\!\left(N(x/n,1),N(0,1)\right) \leq E\!\left[\min\left\{2,\frac{x}{n}\right\}\right] =\min\left\{2,\frac{x}{n}\right\}.

For $x\in[0,1]$ and $n\geq1$, $x/n\leq1$, so the right-hand side is $x/n$. Consequently,

D_n(Q)\leq\int_0^1\frac{x}{n}\,dx=\frac{1}{2n}\longrightarrow0.

This verifies integrated convergence directly. It also gives pointwise weak convergence for every $x$, including the endpoints, because the same distance bound tends to zero for each fixed $x$.

Common Mistake: Treating an Average Guarantee as a Pointwise One

A tempting but incorrect argument is: “The integrated error tends to zero, so the conditional estimate must be accurate at every $x$.” The integral only controls covariate values according to $Q$; sets of small probability can contain large errors, and individual points may have probability zero. The correct interpretation is convergence in $Q$-measure, or equivalently small average bounded-Lipschitz error—not a uniform guarantee over the covariate space.

The reverse confusion is also possible. Convergence at each fixed $x$ is a statement about conditional laws, not automatically a uniform statement over $x$. To obtain integrated convergence from pointwise convergence, require it for $Q$-almost every $x$ and use boundedness of the metric, as above. Neither conclusion should be silently upgraded to uniform convergence over $x$.

Key takeaway: Weak convergence at a specified covariate value is a pointwise claim about conditional laws. Integrated weak convergence is a population-weighted claim, naturally expressed through the bounded-Lipschitz distance. Integrated accuracy allows exceptional covariate values; pointwise accuracy alone does not assert uniform control.

Check Your Understanding

Use the distinction between fixed-covariate and integrated guarantees in your answers.

State the bounded-continuous-test-function definition of weak convergence for conditional laws at a fixed $x$.
Why does $D_n(Q)\to0$ imply that the conditional-law error tends to zero in $Q$-measure?
In the moving-interval example, why can the error at $x=0$ persist even though the integrated error vanishes?
What additional condition allows almost-everywhere pointwise weak convergence to imply integrated convergence?
Does convergence of conditional expectations for one bounded continuous test function establish weak convergence? Explain.