Integral Probability Metric Bounds for Distribution Regression

Why Bound Conditional Distributions Through Test Functions?

A conditional distribution estimate is useful only relative to a way of measuring its error. An integral probability metric (IPM) makes that choice explicit: it compares two distributions by asking how differently they average functions from a specified test class. The choice of class determines what kinds of discrepancies matter, while its size also determines how difficult the estimation problem is.

For distribution regression, the same test functions connect three objects: the conditional law, its conditional expectations (ordinary regression targets), and, for kernel classes, a mean embedding. This tutorial derives bounds that keep those parts separate. The resulting inequalities identify approximation or localization bias, regression or representation error, and a stochastic term governed by test-class complexity.

IPMs and Conditional Regression Targets

The function \(m_f\) is a standard conditional expectation: for each fixed \(f\) with suitable integrability, \(m_f(X)\) is a version of \(E[f(Y)\mid X]\). Consequently, an arbitrary version of that conditional expectation agrees with \(\int f\,dQ_x\) for \(P_X\)-almost every \(x\), not necessarily at every specified query point. When a value at a particular \(x\) is needed, we use the version defined by \(m_f(x):=\int f\,dQ_x\). For a supremum over a class, the versions must be compatible for the functions under consideration; the finite classes below have no ambiguity of this kind. For general classes, measurability of the supremum is also assumed, or outer probability can be used.

To prove the inequality, fix any \(f\in\mathcal F\), add and subtract \(\widehat m_f(x)\), and apply the triangle inequality: \(\left|\int f\,d\widehat Q_x-\int f\,dQ_x\right|\leq \left|\int f\,d\widehat Q_x-\widehat m_f(x)\right|+ \left|\widehat m_f(x)-m_f(x)\right|\). Each term is bounded by its supremum over \(\mathcal F\); taking the supremum of the left side proves the result. \(\square\)

The first term is a compatibility or representation error: does the proposed distribution have the estimated test-function averages? The second is uniform regression error over the test class. If the fitted regression values are exactly the integrals under \(\widehat Q_x\), the first term is zero. The inequality does not require that every regression procedure produce a valid distribution; if it does not, the first term records the mismatch whenever the integrals are defined.

A Finite-Class Bound for Weighted Conditional Empirical Laws

A useful concrete estimator is a weighted empirical distribution. For fixed covariates \(X_1,\ldots,X_n\) and a query point \(x\), take nonnegative weights \(w_i(x)\) with \(\sum_{i=1}^n w_i(x)=1\), and define \(\widehat Q_x=\sum_{i=1}^n w_i(x)\delta_{Y_i}\). Conditional on the covariates, the responses are independent but need not have identical distributions: \(Y_i\) has conditional law \(Q_{X_i}\). The weighted average of those laws is the center \(\bar Q_x=\sum_i w_i(x)Q_{X_i}\). Its difference from \(Q_x\) is a bias term, while \(\widehat Q_x-\bar Q_x\) is the stochastic fluctuation.

Proof. For a fixed \(f\in\mathcal F\), define \(Z_f=\sum_i w_i(x)\{f(Y_i)-E[f(Y_i)\mid X_1,\ldots,X_n]\}\). Conditional independence makes the summands independent and centered. Because \(f(Y_i)\in[-1,1]\), the \(i\)th weighted summand lies in an interval of length \(2w_i(x)\). Hoeffding's inequality therefore gives $$ P\left(|Z_f|\geq t\mid X_1,\ldots,X_n\right) \leq 2\exp\left\{-\frac{t^2}{2\sum_i w_i(x)^2}\right\}. $$

Set \(t=\sqrt{2s_x^2\log(2M/\delta)}\). The right side for each \(f\) is \(\delta/M\). A union bound over the \(M\) functions shows that, with conditional probability at least \(1-\delta\), every \(f\in\mathcal F\) satisfies \(|Z_f|\leq t\). By the definitions, \(Z_f=\int f\,d\widehat Q_x-\int f\,d\bar Q_x\), so \(d_{\mathcal F}(\widehat Q_x,\bar Q_x)\leq t\) on this event. The triangle inequality for the IPM gives \(d_{\mathcal F}(\widehat Q_x,Q_x)\leq d_{\mathcal F}(\widehat Q_x,\bar Q_x)+d_{\mathcal F}(\bar Q_x,Q_x)\), proving the claim. \(\square\)

The quantity \(1/s_x^2\) acts as an effective sample size: with \(n\) equal weights, \(w_i=1/n\), we have \(s_x^2=1/n\). Unequal weights reduce the effective sample size because \(\sum_i w_i^2\) increases when more weight is concentrated on fewer observations. The logarithmic factor records the cost of controlling all \(M\) test functions simultaneously.

The bias term has a useful direct bound when the conditional laws vary smoothly in the chosen metric. If \(d_{\mathcal F}(Q_z,Q_x)\leq L\|z-x\|^\beta\) for the covariate values at hand, then $$ d_{\mathcal F}(\bar Q_x,Q_x) \leq \sum_i w_i(x)d_{\mathcal F}(Q_{X_i},Q_x) \leq L\sum_i w_i(x)\|X_i-x\|^\beta. $$

For the first inequality, fix \(f\), express the difference of its expectations under \(\bar Q_x\) and \(Q_x\) as a weighted sum of differences under \(Q_{X_i}\) and \(Q_x\), and apply the triangle inequality. Bound each summand by the corresponding IPM, then take the supremum over \(f\). This step shows how the metric bias and the weighted stochastic term combine without treating them as the same error.

Regression Error and Kernel Mean Embeddings

The finite-class result can also be read as a uniform regression bound. For each \(f\), the centered weighted sum compares an empirical estimate of the conditional expectation of \(f(Y)\) with its expectation under the weighted conditional law. More generally, if a method directly estimates the regression targets \(m_f(x)\), the deterministic decomposition shows that a uniform bound over \(f\in\mathcal F\) controls the corresponding IPM, provided the estimated distribution's test averages agree with those fitted targets.

A kernel class gives a particularly compact representation. Let \(k\) be a measurable positive-definite kernel with reproducing kernel Hilbert space \(\mathcal H\), and assume \(k(y,y)\leq\kappa^2\) for every \(y\). Its feature map is \(\phi(y)=k(y,\cdot)\), with \(\|\phi(y)\|_{\mathcal H}\leq\kappa\). The kernel mean embedding of a distribution \(P\) is \(\mu_P=E_{Y\sim P}[\phi(Y)]\), defined as a Hilbert-space expectation under these boundedness conditions.

Indeed, the reproducing property gives \(E_P[f(Y)]-E_Q[f(Y)]=\langle f,\mu_P-\mu_Q\rangle_{\mathcal H}\). Cauchy–Schwarz bounds the absolute value by \(\|f\|_{\mathcal H}\|\mu_P-\mu_Q\|_{\mathcal H}\), hence by the embedding norm for every \(f\) in the unit ball. If the difference of embeddings is nonzero, choose \(f=(\mu_P-\mu_Q)/\|\mu_P-\mu_Q\|_{\mathcal H}\); this function belongs to the unit ball and attains that norm. If the difference is zero, both sides are zero. Thus the identity holds. \(\square\)

Consequently, an estimate \(\widehat\mu(x)\) of the conditional embedding \(\mu(x)=\int\phi(y)\,dQ_x(y)\) yields \(d_{\mathcal F}(\widehat Q_x,Q_x)=\|\widehat\mu(x)-\mu(x)\|_{\mathcal H}\) when \(\widehat\mu(x)\) is the embedding of \(\widehat Q_x\). If it is only an approximate embedding, the deterministic decomposition adds the discrepancy between its predicted test-function values and the integrals under the proposed distribution. A characteristic kernel is needed if equality of embeddings is to imply equality of distributions; the displayed error identity itself does not require that implication.

Metric Complexity Beyond Finite Classes

For an infinite test class, a finite covering argument gives a useful version of the same reasoning. Suppose \(\mathcal F\) is bounded by 1 in absolute value and has a finite \(\epsilon\)-net \(\mathcal N_\epsilon\subseteq\mathcal F\) in the uniform norm, with \(N(\epsilon)\) elements. For any \(f\in\mathcal F\), choose \(g\in\mathcal N_\epsilon\) with \(\sup_y|f(y)-g(y)|\leq\epsilon\). For any two probability distributions \(P,Q\), \(\left|(P-Q)f-(P-Q)g\right|\leq |P(f-g)|+|Q(f-g)|\leq2\epsilon\).

It follows that \(d_{\mathcal F}(\widehat Q_x,\bar Q_x)\leq d_{\mathcal N_\epsilon}(\widehat Q_x,\bar Q_x)+2\epsilon\). Applying the finite-class theorem to the net controls the first term with \(M=N(\epsilon)\); the approximation contributes the explicit \(2\epsilon\) remainder. Thus the number of distinguishable test functions at resolution \(\epsilon\), rather than merely whether the class is finite, controls the stochastic cost. This elementary bound can be conservative; sharper bounds require more refined complexity arguments.

Common Mistake: Treating an Almost-Everywhere Identity as Pointwise

• Incorrect reasoning: “Because \(m_f(X)=E[f(Y)\mid X]\), the equality \(m_f(x)=\int f\,dQ_x\) holds at every query point, so its supremum over \(f\) is automatically valid pointwise.”
• Correction: For each fixed \(f\), a conditional-expectation version agrees with \(\int f\,dQ_x\) for \(P_X\)-almost every \(x\). At a specified query point, use the version defined by \(m_f(x):=\int f\,dQ_x\), or explicitly impose the required pointwise compatibility. For an uncountable class, separate almost-everywhere statements for each function do not by themselves justify one common null set or a measurable supremum.

A related modeling pitfall is to report a stochastic error bound while omitting the distance from the weighted mixture \(\bar Q_x\) to the target \(Q_x\). Concentrating weights near \(x\) can reduce this bias under suitable regularity, but the stochastic term depends on the squared weights and the test-class complexity. Both terms are needed for a distributional guarantee.