Calibrating Distributional Ambiguity Radii

Why Calibrate an Ambiguity Radius?

A distributionally robust risk takes a worst case over a neighborhood of a reference distribution. The radius of that neighborhood is not merely a numerical tuning parameter: it determines which alternative distributions the analysis treats as plausible. If the radius is too small, the neighborhood may exclude the data-generating distribution. If it is too large, the resulting worst-case risk can be needlessly conservative.

A principled calibration connects the radius to sampling uncertainty. Given observations from an unknown distribution \(P\), construct a set that contains \(P\) with probability at least \(1-\alpha\), where \(\alpha\) is a chosen error probability. This gives a coverage interpretation for the radius. It does not, by itself, guarantee protection against a separate shift from \(P\) to a target distribution; that requires assumptions about the shift as well.

The last point matters especially for divergences, which are not generally symmetric. A finite-sample guarantee for \(D_{\mathrm{KL}}(\widehat P_n\Vert P)\) does not automatically give the same guarantee for \(D_{\mathrm{KL}}(P\Vert\widehat P_n)\), or for a robust-risk set using the reverse orientation. Wasserstein distance, by contrast, is a metric and is symmetric.

Finite-Sample Calibration With Divergences

Suppose each observation takes one of \(m\) possible values, and let \(P\) have probability vector \(p=(p_1,\ldots,p_m)\). From \(n\) independent observations drawn from \(P\), let \(\widehat p_n\) be the empirical probability vector. For probability vectors \(q,p\), define the Kullback–Leibler divergence by

with value \(+\infty\) if \(q_j>0\) but \(p_j=0\) for some \(j\). A finite-sample coverage bound is available for the orientation from empirical distribution to true distribution.

Proof. A possible empirical probability vector \(q\) is a type: its coordinates are integer multiples of \(1/n\). There are at most \((n+1)^m\) such vectors. For any one type \(q\), the probability of observing that type under \(p\) is at most \(\exp\{-nD_{\mathrm{KL}}(q\Vert p)\}\). To see the bound, write the probability as the number of sequences of type \(q\) times their common probability under \(p\). The number of such sequences is at most \(\exp\{nH(q)\}\), where \(H(q)=-\sum_jq_j\log q_j\); each sequence has probability \(\exp\{n\sum_jq_j\log p_j\}\). Multiplying gives the claimed upper bound. If \(q_j>0\) where \(p_j=0\), that type instead has probability zero.

For any \(\varepsilon>0\), sum this bound over all types with \(D_{\mathrm{KL}}(q\Vert p)>\varepsilon\). There are at most \((n+1)^m\) types, so the probability of this event is at most \((n+1)^m e^{-n\varepsilon}\). Substituting \(\varepsilon=[m\log(n+1)+\log(1/\alpha)]/n\) makes this upper bound equal to \(\alpha\). \(\square\)

This theorem is finite-sample and distribution-free over the finite alphabet, but its radius can be conservative, particularly when \(m\) is large relative to \(n\). It also calibrates the set \(\{Q:D_{\mathrm{KL}}(\widehat P_n\Vert Q)\leq\rho\}\), not automatically the commonly used reverse-oriented set \(\{Q:D_{\mathrm{KL}}(Q\Vert\widehat P_n)\leq\rho\}\). The empirical distribution assigns zero mass to unobserved categories, so the reverse orientation can exclude a true distribution that gives those categories positive probability, regardless of how small a finite radius is chosen.

Total Variation as a Direct Probability-Simplex Bound

A convenient alternative on a finite alphabet is total variation distance, defined by \(\operatorname{TV}(q,p)=\frac12\sum_{j=1}^m|q_j-p_j|\). It also equals \(\sup_A|q(A)-p(A)|\), where \(A\) ranges over subsets of the alphabet. This identity allows a direct concentration argument.

Derivation. For a fixed subset \(A\), the empirical probability \(\widehat P_n(A)\) is an average of \(n\) independent indicator variables with mean \(P(A)\). Hoeffding's inequality gives \(P\{|\widehat P_n(A)-P(A)|>t\}\leq2e^{-2nt^2}\). There are \(2^m\) subsets. Since total variation is the largest absolute discrepancy over these subsets, the union bound gives

Setting the right-hand side equal to \(\alpha\) and solving for \(t\) yields the stated radius. This argument is conservative because it controls all subsets simultaneously, but it does not require estimating a particular divergence or its orientation.

Wasserstein Radii on a Finite Metric Space

Wasserstein calibration is useful when the geometry between outcomes matters. Let the possible outcomes form a finite metric space with distance \(d\) and diameter \(D=\max_{x,y}d(x,y)\). The 1-Wasserstein distance \(W_1(Q,P)\) is the infimum, over joint distributions with marginals \(Q\) and \(P\), of the expected distance between the paired outcomes.

Proof. First, transporting the common mass at each point costs zero. The remaining unmatched mass is \(\operatorname{TV}(Q,P)\), and it can be transported at cost at most \(D\) per unit mass. Therefore \(W_1(Q,P)\leq D\operatorname{TV}(Q,P)\).

Write \(p_j=P(\{x_j\})\) and \(\widehat p_j=\widehat P_n(\{x_j\})\). By the variance bound for empirical proportions and the Cauchy–Schwarz inequality,

The last inequality follows from \((\sum_j\sqrt{p_j})^2\leq m\sum_jp_j=m\). Hence \(E[W_1(\widehat P_n,P)]\leq D\sqrt{m/n}/2\). Replacing one observation changes the empirical measure by moving mass \(1/n\) between two points, so it changes \(W_1(\widehat P_n,P)\) by at most \(D/n\), by the triangle inequality. McDiarmid's inequality therefore gives \(P\{W_1-EW_1>t\}\leq\exp(-2nt^2/D^2)\). Taking \(t=D\sqrt{\log(1/\alpha)/(2n)}\) and combining with the expectation bound proves the result. \(\square\)

This bound makes the role of geometry explicit through \(D\), but it can be loose: it uses only the diameter and number of support points. Sharper rates may be available with additional structure. In particular, a finite-support guarantee should not be presented as a dimension-free guarantee for an arbitrary continuous distribution.

From Coverage to Worst-Case Risk

A coverage-calibrated set provides a statistical reason to consider its members in a robust risk calculation. For a loss function \(\ell\) taking values in \([0,1]\), the difference in expected loss under two distributions is bounded by their total variation distance:

Consequently, if a total-variation ball around \(\widehat P_n\) contains \(P\) with probability at least \(1-\alpha\), then the worst-case expected loss over that ball is, with the same probability, at least \(E_P[\ell]\). For a loss that is \(L\)-Lipschitz with respect to the ground metric, the coupling definition gives \(|E_Q[\ell]-E_P[\ell]|\leq L W_1(Q,P)\). Thus a Wasserstein radius also translates into a risk allowance, scaled by \(L\).

The practical trade-off is direct. Increasing \(\rho\) enlarges the set over which the risk is maximized, so the worst-case risk cannot decrease. A larger radius can improve protection against sampling uncertainty or plausible distributional deviations, but it may make the reported risk less representative of nominal performance. A smaller radius can yield a more efficient nominal assessment while increasing the chance that the true distribution lies outside the set. The confidence level controls one part of this choice: smaller \(\alpha\) requires a larger coverage radius.

In practice, report the reference distribution, distance and orientation, sample size, support or metric assumptions, confidence level, and resulting radius. Then examine how the worst-case risk changes over a range of radii. This sensitivity analysis makes visible whether conclusions depend on a narrow nominal neighborhood or remain stable under stronger protection.