Source And Target Risks Under Distribution Shift

Why Source Risk May Not Predict Target Risk

A learning algorithm is often trained and evaluated on data from one population, then used in another. Even if the predictor is held fixed, its expected loss can change because the distribution of features, outcomes, or their relationship has changed. The key question is not simply whether a shift has occurred, but which part of the joint distribution changed and what information remains available to calculate target risk.

We will define risks relative to source and target distributions, then distinguish covariate shift, label shift, and conditional shift. These names describe different assumptions about how the joint distribution changes; they are not interchangeable. In particular, a risk-transfer formula is valid only when its invariance and support conditions hold.

Definition: Let $(X,Y)$ take values in feature and outcome spaces $\mathcal X$ and $\mathcal Y$. A source distribution $P_s$ and a target distribution $P_t$ are probability distributions for the pair $(X,Y)$. For a fixed predictor $h:\mathcal X\to\mathcal A$ and measurable loss $\ell:\mathcal A\times\mathcal Y\to[0,\infty)$, define the source and target risks by $$ R_s(h)=E_{P_s}[\ell(h(X),Y)],\qquad R_t(h)=E_{P_t}[\ell(h(X),Y)]. $$ Assume these expectations exist and are finite. The predictor and loss are held fixed when comparing the two risks.

For classification with $\mathcal A=\mathcal Y$, a standard choice is zero-one loss, $\ell(a,y)=\mathbf 1\{a\ne y\}$. For regression, squared or absolute error may be used when the corresponding expectations are finite. The definition of risk does not depend on a particular loss, but any risk-transfer claim does depend on using the same loss under both distributions.

Three Ways the Joint Distribution Can Change

Write $P_s^X$ and $P_t^X$ for the source and target feature marginals, and $P_s^Y$ and $P_t^Y$ for the outcome marginals. Conditional distributions such as $P_s(Y\mid X)$ describe the outcome mechanism at a given feature value. The following names specify which of these components is assumed invariant.

Shift type	Assumption	What may change
Covariate shift	$P_t(Y\mid X)=P_s(Y\mid X)$	The feature marginal $P^X$
Label shift (prior probability shift)	$P_t(X\mid Y)=P_s(X\mid Y)$	The outcome marginal $P^Y$, including class proportions
Conditional shift	$P_t(Y\mid X)\ne P_s(Y\mid X)$ on a set of positive target feature probability	The outcome mechanism given features

These descriptions concern probability distributions, not just changes in sample averages. A finite sample can show different empirical proportions by chance even when the population distributions agree. Also, the labels are not a complete taxonomy: a shift can change more than one component at once. If both the feature marginal and the conditional outcome distribution change, neither pure covariate-shift nor pure label-shift formulas can be applied without further assumptions.

Risk Transfer Under Covariate Shift

Under covariate shift, the conditional distribution of $Y$ given $X$ is invariant. The feature distribution can change, so the same conditional prediction performance is averaged over a different set of feature values. A common support condition is essential: every feature region with positive target probability must also be represented under the source distribution.

Theorem (covariate-shift risk identity): Suppose $P_s(Y\mid X)=P_t(Y\mid X)$ for $P_t^X$-almost every $x$, the loss is nonnegative and measurable, and $P_t^X$ is absolutely continuous with respect to $P_s^X$, written $P_t^X\ll P_s^X$. Let $w=dP_t^X/dP_s^X$ be the Radon–Nikodym derivative. Then $$ R_t(h)=E_{P_s}\!\left[w(X)\ell(h(X),Y)\right]. $$ This equality holds for nonnegative risks, including the possibility that either side is infinite; if the target risk is finite, both sides are finite and equal.

To prove the identity, define the conditional expected loss $g(x)=E[\ell(h(x),Y)\mid X=x]$ using a version of the common conditional distribution. By iterated expectation, the target risk is the integral of $g$ against the target feature marginal. Absolute continuity and the defining property of the Radon–Nikodym derivative give the change of measure:

\begin{aligned} R_t(h) &=\int g(x)\,P_t^X(dx)\\ &=\int g(x)w(x)\,P_s^X(dx)\\ &=\int w(x)g(x)\,P_s^X(dx)\\ &=E_{P_s}\!\left[w(X)\ell(h(X),Y)\right]. \end{aligned}

The last equality follows by conditioning on $X$ under the source distribution: $w(X)$ is a function of $X$, and the source conditional expected loss is $g(X)$. This derivation explains both the assumption and the limitation. Invariance supplies the same conditional loss function, while absolute continuity rules out target-only feature regions whose conditional losses cannot be learned from the source distribution.

Worked Example: Covariate Shift With Two Feature Values

Let $X\in\{a,b\}$ and use zero-one loss. Under the source distribution, $P_s(X=a)=0.8$ and $P_s(X=b)=0.2$. Under the target distribution, $P_t(X=a)=0.3$ and $P_t(X=b)=0.7$. Suppose the conditional error probabilities of a fixed classifier are $P(Y\ne h(X)\mid X=a)=0.1$ and $P(Y\ne h(X)\mid X=b)=0.4$ in both populations.

The source risk is the source-weighted average of the two conditional error rates: $$ R_s(h)=0.8(0.1)+0.2(0.4)=0.08+0.08=0.16. $$ The target risk uses the target feature probabilities: $$ R_t(h)=0.3(0.1)+0.7(0.4)=0.03+0.28=0.31. $$

Both feature values have positive probability in the source, so the support condition holds. The density ratios are $w(a)=0.3/0.8=0.375$ and $w(b)=0.7/0.2=3.5$. Weighting source feature contributions gives $$ 0.8(0.375)(0.1)+0.2(3.5)(0.4)=0.03+0.28=0.31, $$ which agrees with the direct target calculation. The conditional performance did not worsen at either feature value; target risk rose because the target population places more mass on the harder value $b$.

Risk Transfer Under Label Shift

For a finite outcome space, label shift assumes that the feature distribution within each class is unchanged, while class probabilities may differ. Define the class-conditional risk $r_s(y)=E_{P_s}[\ell(h(X),y)\mid Y=y]$ whenever $P_s(Y=y)>0$. Under label shift, the conditional distribution of $X$ given $Y=y$ is the same in source and target for classes where these conditionals are defined.

Theorem (label-shift risk decomposition): Let $\mathcal Y$ be finite. Suppose $P_t(X\mid Y=y)=P_s(X\mid Y=y)$ for every class with $P_t(Y=y)>0$, and suppose each such class also satisfies $P_s(Y=y)>0$. Then $$ R_t(h)=\sum_{y:\,P_t(Y=y)>0}P_t(Y=y)\,r_s(y). $$ The sum is explicitly restricted to target-present classes, so it contains no undefined class-conditional risk for a class with zero source probability.

For each target-present class, the target conditional expected loss equals the source class-conditional expected loss: the predictor and loss are fixed, and the distribution of $X\mid Y=y$ is invariant. Applying the law of total expectation under the target distribution therefore gives

\begin{aligned} R_t(h) &=\sum_{y:\,P_t(Y=y)>0}P_t(Y=y) E_{P_t}[\ell(h(X),y)\mid Y=y]\\ &=\sum_{y:\,P_t(Y=y)>0}P_t(Y=y) E_{P_s}[\ell(h(X),y)\mid Y=y]\\ &=\sum_{y:\,P_t(Y=y)>0}P_t(Y=y)r_s(y). \end{aligned}

The source-positive condition is substantive. If a class has positive target probability but zero source probability, its source class-conditional risk is not defined from the source distribution, and label-shift invariance cannot supply information about a source conditional distribution that has no source class observations. The formula is not justified for that class without additional information or assumptions.

Worked Example: Changing Class Proportions

Consider binary classification with classes $0$ and $1$. Suppose the source class probabilities are $P_s(Y=0)=0.8$ and $P_s(Y=1)=0.2$. A fixed classifier has class-conditional error rates $r_s(0)=0.05$ and $r_s(1)=0.30$. Under label shift, suppose the target class probabilities are $P_t(Y=0)=0.4$ and $P_t(Y=1)=0.6$, with both class-conditional feature distributions unchanged.

The source risk is $$ R_s(h)=0.8(0.05)+0.2(0.30)=0.04+0.06=0.10. $$ The target risk is $$ R_t(h)=0.4(0.05)+0.6(0.30)=0.02+0.18=0.20. $$ Both classes are present in both populations, so the class-conditional risks used in the target sum are defined. The target risk doubles even though the class-specific error rates stay fixed: the target population contains a larger proportion of the class on which this classifier makes more errors.

When Conditional Shift Prevents Risk Recovery

Under conditional shift, $P(Y\mid X)$ changes. A source risk is an average under the source joint distribution and does not, by itself, reveal the target conditional loss. Even if the feature marginal is unchanged, different target outcome mechanisms can produce different target risks while leaving all source information the same.

Proposition (source information alone need not identify target risk): There exist two target distributions with the same source distribution and the same target feature marginal, but different target risks for the same predictor and loss. Consequently, source data and the target feature marginal alone do not generally determine target risk when the conditional outcome distribution is unrestricted.

To verify the claim, let $X$ be constant, let $Y\in\{0,1\}$, use predictor $h(X)=0$, and use zero-one loss. Let the source distribution have $P_s(Y=1)=0.1$. Consider two possible target distributions, both with the same constant feature value: in the first, $P_t(Y=1)=0.1$; in the second, $P_t(Y=1)=0.9$. The source distribution and target feature marginal are identical across these two scenarios. Yet the target risks are respectively $0.1$ and $0.9$, because the classifier predicts zero and errs exactly when $Y=1$. Thus the available source and feature-marginal information cannot distinguish these target risks. $\square$

Worked Example: Detecting Unrecoverable Conditional Change

Suppose $X$ takes only the value $x_0$. In the source, $P_s(Y=1\mid X=x_0)=0.2$. A classifier predicts $h(x_0)=0$, and the loss is zero-one loss. Compare two possible target mechanisms: $P_t(Y=1\mid X=x_0)=0.2$ or $P_t(Y=1\mid X=x_0)=0.7$.

Since the classifier predicts zero, it errs precisely for outcomes with $Y=1$. Its source risk is $R_s(h)=0.2$. Under the first target mechanism, $R_t(h)=0.2$; under the second, $R_t(h)=0.7$. The feature distribution is unchanged because $X=x_0$ with probability one in all cases, but the conditional outcome distribution differs. Therefore covariate-shift reasoning is inapplicable, and knowing only the unchanged feature marginal does not recover the target risk.

Assumptions Are Part of the Risk Claim

A risk-transfer equation is not just algebra. It encodes what must remain stable between populations and what information must be available. Covariate shift uses a common conditional outcome mechanism and adequate feature support. Label shift uses common class-conditional feature distributions and requires source observations for every target-present class. Conditional shift generally requires additional target outcome information or stronger structural assumptions.

A frequent mistake is to say, “The feature distributions differ, so reweight the source risk,” without checking whether the conditional distribution $P(Y\mid X)$ is invariant and whether the target feature distribution is supported by the source. Another is to use a label-shift sum over every class, including a class with zero target probability whose source conditional risk is undefined. Write the sum only over target-present classes, and verify that each such class is source-present before using its source class-conditional risk.

These distinctions matter whenever performance is reported for a population different from the one that generated the training data. A low source risk is not automatically evidence of low target risk. The relevant question is whether the assumed invariances connect the source and target expectations, and whether the required parts of the target distribution can be learned or are otherwise known.

Key takeaway: Source and target risks are expectations under different joint distributions. Covariate shift and label shift permit different risk-transfer decompositions under distinct invariance and support assumptions; conditional shift generally leaves target risk unidentified from source risk alone.

Check Your Understanding

For each question, state the distributional assumption needed before giving a risk-transfer conclusion.

Under covariate shift, why is $P_t^X\ll P_s^X$ needed for the source-weighted risk identity?
A target population has a class with positive probability that never occurs in the source. Why can its source class-conditional risk not be inserted into the label-shift formula?
In the two-feature worked example, which feature value accounts for the increase from source risk to target risk?
Can an unchanged feature marginal alone guarantee unchanged risk? Explain using the conditional distribution of $Y\mid X$.
State the invariance assumption for label shift, and distinguish it from the invariance assumption for covariate shift.

Shift type	Assumption	What may change
Covariate shift	\(P_t(Y\mid X)=P_s(Y\mid X)\)	The feature marginal \(P^X\)
Label shift (prior probability shift)	\(P_t(X\mid Y)=P_s(X\mid Y)\)	The outcome marginal \(P^Y\), including class proportions
Conditional shift	\(P_t(Y\mid X)\ne P_s(Y\mid X)\) on a set of positive target feature probability	The outcome mechanism given features