Comparing Weighting And Worst-Case Risk Transfer

Three Different Questions About Risk Transfer

Suppose a predictor $h$ is evaluated under a source distribution $P$, but will be used under a target distribution $Q$. Write $Z=(X,Y)$, and let $\ell(h(X),Y)$ be a nonnegative measurable loss. The target risk is

R_Q(h)=E_Q[\ell(h(X),Y)].

Three approaches can relate source information to target performance, but they answer different questions. Importance weighting estimates the target risk by changing the contribution of source observations. Divergence-based distributionally robust optimization bounds risk over a specified set of possible distributions. Adversarial risk evaluates the worst loss after allowed input perturbations. A formula that is valid for one of these targets is not automatically a guarantee for the others.

The comparison is easiest when each method is stated in terms of its estimand, assumptions, and guarantee. A guarantee is always conditional on the model of shift being correct: for example, on the target being absolutely continuous with respect to the source, lying inside a divergence ball, or being reachable by allowed perturbations.

Importance Weighting: Estimate a Specified Target Risk

Definition: Under covariate shift, the source and target conditional distributions of $Y$ given $X$ agree, while the marginal distribution of $X$ may differ: $Q(Y\mid X)=P(Y\mid X)$. If $Q_X$ is absolutely continuous with respect to $P_X$, define the density ratio $w(x)=dQ_X/dP_X(x)$. The importance-weighted source risk is $E_P[w(X)\ell(h(X),Y)]$.

Conditional invariance says that the relationship between features and outcomes is unchanged; only the frequency of feature values changes. Absolute continuity, $Q_X\ll P_X$, means that any feature set assigned probability zero by the source also has target probability zero. This support condition is essential: a source sample cannot reveal the loss in a region it never contains.

Theorem: If $Q(Y\mid X)=P(Y\mid X)$, $Q_X\ll P_X$, and $\ell(h(X),Y)$ is integrable under $Q$, then $$ R_Q(h)=E_P[w(X)\ell(h(X),Y)]. $$

Proof. Let $g(x)=E_P[\ell(h(x),Y)\mid X=x]$. Conditional invariance gives the same conditional expectation under $Q$, for $Q_X$-almost every $x$. Therefore,

R_Q(h)=\int g(x)\,dQ_X(x) =\int g(x)w(x)\,dP_X(x) =E_P[w(X)\ell(h(X),Y)].

The second equality is the change-of-measure identity for the Radon–Nikodym derivative; the last follows by conditioning on $X$ under $P$. The assumptions ensure the target integral is defined and that the conditional equality applies on the target support. $\square$

This is an identity for the population target risk, not a claim that its finite-sample estimate is stable. Large or highly variable weights can make estimation noisy. If the conditional outcome law changes, weighting by $X$ alone generally does not repair the mismatch.

Worked Example: Reweighting Under a Known Covariate Shift

Let $X\in\{0,1\}$. Under the source, $P(X=0)=P(X=1)=0.5$; under the target, $Q(X=0)=0.2$ and $Q(X=1)=0.8$. The conditional error probabilities of a fixed classifier are $P(\text{error}\mid X=0)=0.1$ and $P(\text{error}\mid X=1)=0.4$, and assume these conditional probabilities are unchanged in the target.

The target-to-source density ratios are $w(0)=0.2/0.5=0.4$ and $w(1)=0.8/0.5=1.6$. The source risk is $0.5(0.1)+0.5(0.4)=0.25$, which is not the target risk. The weighted source risk is

$$ 0.5(0.4)(0.1)+0.5(1.6)(0.4) =0.02+0.32=0.34. $$

Direct target calculation gives $0.2(0.1)+0.8(0.4)=0.02+0.32=0.34$, agreeing with the identity. The example succeeds because the conditional error rates are invariant and both target feature values have positive source probability.

Divergence Robustness: Protect Against a Set of Distributions

A divergence-based robust risk does not, by itself, identify one target distribution. Given a divergence $D$ and radius $\rho$, an ambiguity set around $P$ can be written as

\mathcal{U}(P,\rho)=\{Q:D(Q\Vert P)\leq \rho\}, \qquad R_{\mathrm{rob}}(h)=\sup_{Q\in\mathcal{U}(P,\rho)}R_Q(h).

The precise meaning depends on the chosen divergence and on which distributions are allowed in the set. The robust risk is a worst-case value, not an estimate of the risk under a particular target. Its population guarantee is direct:

Theorem: For any target distribution $Q$ in $\mathcal{U}(P,\rho)$ for which the risk is defined, $R_Q(h)\leq R_{\mathrm{rob}}(h)$.

Proof. The supremum of a function over a set is at least its value at every member of that set. Since $Q\in\mathcal{U}(P,\rho)$, $R_Q(h)\leq\sup_{Q'\in\mathcal{U}(P,\rho)}R_{Q'}(h)=R_{\mathrm{rob}}(h)$. $\square$

The guarantee is only as useful as the ambiguity set. If the target lies outside it, this theorem says nothing about the target risk. If the set is very broad, its worst-case risk can be conservative. When the set is estimated from data, a population inclusion statement also needs a separate argument that the estimated set contains the target with adequate probability.

Worked Example: A Total-Variation Ball Gives a Worst-Case Bound

Let $X\in\{0,1\}$, with source probabilities $P(X=0)=P(X=1)=0.5$. Suppose the loss of a fixed predictor is $0$ at $X=0$ and $1$ at $X=1$. Consider all distributions on these two points whose total variation distance from $P$ is at most $0.4$, where total variation is $\sup_A|Q(A)-P(A)|$.

Write $q=Q(X=1)$. On a two-point space, the total variation distance is $|q-0.5|$, so the constraint is $0.1\leq q\leq0.9$. The risk under $Q$ is $R_Q(h)=q$, because the loss is one only at $X=1$. Hence the robust risk is

R_{\mathrm{rob}}(h)=\sup_{0.1\leq q\leq0.9}q=0.9.

If the actual target has $Q(X=1)=0.8$, its risk is $0.8$, which is below the robust value $0.9$. If instead the target has $Q(X=1)=0.99$, its risk is $0.99$, and the ball does not contain it: its total variation distance from the source is $0.49$. The calculated robust risk is not a guarantee for this excluded target.

Adversarial Risk: Protect Against Allowed Input Changes

Let $\Delta(x)$ be a set of allowed perturbed inputs for an input $x$. For a loss evaluated with the original label held fixed, adversarial risk under the source is

R_{\mathrm{adv}}(h;P) =E_P\left[\sup_{x'\in\Delta(X)}\ell(h(x'),Y)\right].

This quantity asks how badly the predictor can perform after each source input is perturbed within its allowed set. To interpret it as a transfer guarantee, a target observation must be reachable from a source observation under the specified perturbation rule, with its label preserved. This is a different assumption from conditional invariance or membership in a divergence ball.

Theorem: Suppose there is a coupling $\pi$ of source $(X,Y)\sim P$ and target $(X',Y')\sim Q$ such that $Y'=Y$ and $X'\in\Delta(X)$ almost surely. If the pointwise supremum in $R_{\mathrm{adv}}$ is measurable and integrable, then $R_Q(h)\leq R_{\mathrm{adv}}(h;P)$.

Proof. Under the stated coupling, $Y'=Y$ and $X'\in\Delta(X)$ almost surely. Thus, pointwise under $\pi$,

\ell(h(X'),Y') =\ell(h(X'),Y) \leq \sup_{x'\in\Delta(X)}\ell(h(x'),Y).

Taking expectations under $\pi$, the left side is $E_Q[\ell(h(X'),Y')]=R_Q(h)$, since the target marginal of the coupling is $Q$. The right side is $E_P[\sup_{x'\in\Delta(X)}\ell(h(x'),Y)]=R_{\mathrm{adv}}(h;P)$, since its source marginal is $P$. The inequality follows. $\square$

This result uses a specific reachability condition. It does not say that adversarial risk bounds performance under every change in the feature distribution. The perturbation set must match the shift, and the label-preserving interpretation must be appropriate for the application.

Worked Example: A Shift Outside the Perturbation Set

Let $X$ be real-valued, let $Y\in\{0,1\}$, and use the classifier $h(x)=\mathbf{1}\{x\geq1\}$ with zero-one loss. Under the source, half the observations are $(X,Y)=(0,0)$ and half are $(2,1)$. Let $\Delta(x)=[x-0.25,x+0.25]$.

At source input $0$, every allowed perturbation lies in $[-0.25,0.25]$, where the classifier predicts $0$, so the worst loss for label $0$ is $0$. At source input $2$, every allowed perturbation lies in $[1.75,2.25]$, where the classifier predicts $1$, so the worst loss for label $1$ is also $0$. Therefore $R_{\mathrm{adv}}(h;P)=0$.

Now define a target that moves the first half of the observations to $(1.1,0)$ and leaves the second half at $(2,1)$. At $X=1.1$, the classifier predicts $1$ despite label $0$, giving loss $1$; the other half has loss $0$. Thus $R_Q(h)=0.5$. The moved input is $1.1$, which is $1.1$ units from its source input $0$, outside the allowed radius $0.25$. The coupling condition fails, so the zero adversarial risk does not protect against this target shift.

How to Choose and Interpret the Guarantee

Method	What it targets	Key identifying assumption	What the guarantee says
Importance weighting	Risk under a specified target marginal	Invariant $Y\mid X$, and target feature support contained in source support	The population weighted risk equals the target risk
Divergence robustness	Worst risk over an ambiguity set	The relevant target lies in the chosen set	The robust risk upper-bounds every risk in that set
Adversarial risk	Worst loss over allowed input perturbations	Target inputs are reachable by allowed, label-preserving perturbations	The adversarial risk upper-bounds risk for such a coupled target

The methods may give different numerical answers because their estimands differ. Weighting asks for risk under one target distribution. Divergence robustness takes a supremum over a family of distributions. Adversarial risk takes a pointwise supremum over perturbations before averaging under the source. None is universally preferable: the relevant question is which assumption best describes the shift and which guarantee is needed.

Common Mistake: Treating Robustness as an Assumption-Free Bound

A common but incorrect argument is: “The adversarial risk is large enough to cover perturbations, so it must also cover any distribution shift,” or, similarly, “A divergence-robust risk is worst-case, so it must bound the target risk.” Both conclusions omit the set-membership condition. A worst-case value bounds only the distributions or perturbations over which its supremum is taken.

The correction is to name the target mechanism and verify its condition. For importance weighting, check conditional invariance and support. For divergence robustness, check that the target belongs to the chosen ambiguity set. For adversarial risk, check that target inputs can be coupled to source inputs through the specified perturbations without changing labels. If the condition is doubtful, the reported risk is not a guaranteed bound for that target.

Key takeaway: Always report both the risk quantity and the assumption linking it to deployment. A target-risk estimate, an ambiguity-set worst case, and a local perturbation worst case are distinct objects, even when all are computed from the same source data.

Check Your Understanding

For each question, state the assumption that makes the proposed risk calculation relevant to the target.

Under what two conditions does importance weighting recover target risk from source data?
In the total-variation example, why does a target with $Q(X=1)=0.99$ escape the robust-risk guarantee?
What additional condition on a source-target coupling makes adversarial risk an upper bound for target risk?
Can a low adversarial risk for radius $0.25$ guarantee low target risk when target inputs move by $1.1$? Explain.
Which method targets one specified distributional risk, and which two are based on worst-case quantities?

Method	What it targets	Key identifying assumption	What the guarantee says
Importance weighting	Risk under a specified target marginal	Invariant \(Y\mid X\), and target feature support contained in source support	The population weighted risk equals the target risk
Divergence robustness	Worst risk over an ambiguity set	The relevant target lies in the chosen set	The robust risk upper-bounds every risk in that set
Adversarial risk	Worst loss over allowed input perturbations	Target inputs are reachable by allowed, label-preserving perturbations	The adversarial risk upper-bounds risk for such a coupled target