Worked Risk Transfer Under Multiple Shift Assumptions

One Prediction Problem, Several Transfer Assumptions

A risk estimate for a target population is not determined by source labels and unlabeled target covariates alone. It also depends on what relationship between source and target distributions is assumed. In this tutorial, a fixed binary classifier is evaluated using three calculations: covariate-shift weighting, label-shift correction, and a robust worst case over both models. The point is not that one estimate is automatically preferable, but that each estimate answers a different conditional question.

Let \(X\) be a binary covariate with values \(a\) and \(b\), and let \(Y\in\{0,1\}\) be the label. A fixed classifier \(g\) predicts a label from \(X\). We use the zero-one loss \(\ell(g(X),Y)=\mathbf{1}\{g(X)\ne Y\}\), so the target risk is \(R_T=E_T[\ell(g(X),Y)]\). The source distribution \(P_S\) supplies labeled observations; the target distribution \(P_T\) supplies covariate observations but no labels.

The Source Data and Target Covariates

Suppose the source sample has 1,000 observations, summarized by the following counts. The classifier predicts 0 when \(X=a\), and 1 when \(X=b\). Thus errors occur in the \(Y=1,X=a\) cell and the \(Y=0,X=b\) cell.

The source risk is \((300+400)/1000=0.70\). The source covariate probabilities are \(P_S(X=a)=P_S(X=b)=0.5\). The conditional error probabilities, estimated here by the corresponding source proportions, are \(e_a=P_S(\ell=1\mid X=a)=300/500=0.6\) and \(e_b=P_S(\ell=1\mid X=b)=400/500=0.8\). The target sample has 100 covariate observations: 70 are \(a\) and 30 are \(b\). Its empirical covariate proportions are therefore 0.7 and 0.3.

Why Covariate Weighting Works Under Its Assumption

The population identity behind the calculation is a change of measure. Write \(r(x)=p_T(x)/p_S(x)\) when densities exist, or use the corresponding Radon–Nikodym derivative for general distributions. The support condition means \(P_T\) is absolutely continuous with respect to \(P_S\) on \(X\), so this ratio is defined \(P_T\)-almost surely.

To verify the identity, let \(m_S(x)=E_S[\ell(g(X),Y)\mid X=x]\) and define \(m_T\) analogously. Equality of the conditional distributions implies \(m_T(x)=m_S(x)\) for \(P_T\)-almost every \(x\). Therefore

The last equality follows by conditioning the source expectation on \(X\). In the finite binary example, the integral is just the sum over \(a\) and \(b\), producing the two weighted error contributions above. Large or unstable ratios can make the sample estimate variable even when the identity is correct.

Label-Shift Correction from Unlabeled Covariates

Under label shift, the class-conditional distribution of \(X\) transfers instead. From the source table, \(P_S(X=a\mid Y=1)=300/400=0.75\), while \(P_S(X=a\mid Y=0)=200/600=1/3\). Let \(\pi_T=P_T(Y=1)\). If label shift holds, the target marginal probability of \(a\) satisfies \(P_T(X=a)=0.75\pi_T+(1/3)(1-\pi_T)\). Thus an unlabeled target covariate proportion can identify the target class prevalence here because the two source class-conditional probabilities differ.

The label-shift risk calculation also has a direct derivation. Under label shift, condition first on \(Y\): the conditional expected loss given each class is the same in source and target. If \(r_y=E_S[\ell\mid Y=y]\) and \(\pi_T=P_T(Y=1)\), then

For prevalence identification, put \(q_a=P_T(X=a)\), \(u_1=P_S(X=a\mid Y=1)\), and \(u_0=P_S(X=a\mid Y=0)\). The label-shift equation is \(q_a=u_0+(u_1-u_0)\pi_T\). If \(u_1\ne u_0\), its unique solution is \(\pi_T=(q_a-u_0)/(u_1-u_0)\). A solution outside \([0,1]\) signals incompatibility between the observed marginal and this fitted label-shift model, rather than a valid prevalence estimate. Here \(u_1=0.75\ne 1/3=u_0\), and the solution 0.88 is in range.

A Robust Estimate Across Competing Models

A robust estimate can protect against uncertainty about which transfer model is appropriate, but it is only meaningful after specifying the candidate models and the uncertainty set. For this example, use an approximate 95% Wald interval for the target proportion \(q_a\). From 70 occurrences of \(a\) in 100 independent target observations, \(\hat q_a=0.70\) and its estimated standard error is \(\sqrt{0.7(0.3)/100}=0.0458\). The interval is \(0.70\pm1.96(0.0458)\), approximately \([0.610,0.790]\). This interval accounts for target covariate sampling only, not for source estimation or model misspecification.

This robust value is not a third fitted description of the observed target population. It is an upper bound over the specified set of possibilities. Its value depends on the interval, on which shift mechanisms are included, and on the transferred source rates being treated as fixed. A broader or differently constructed uncertainty set can yield a different worst case.

Uncertainty, Assumptions, and a Common Pitfall

There are at least three uncertainty sources to keep separate. First, the target proportion \(q_a\) is estimated from a finite unlabeled sample; the interval above addresses this component approximately. Second, source conditional error rates and class-conditional covariate probabilities are estimated from finite labeled data. Third, the choice of shift assumption is structural: sampling more target covariates does not by itself determine whether covariate shift or label shift holds. The robust calculation addresses the third issue only to the extent that its model set contains the plausible mechanisms.

A frequent mistake is to say, “Target covariates are observed, so reweighting gives the target risk.” Reweighting gives the target risk only if the conditional label distribution given \(X\) transfers and the required support condition holds. In this example, reweighting gives 0.66 under covariate shift, while label shift gives 0.74 from the same observed marginal. Unlabeled covariates alone do not reveal which conditional relationship is true. A second pitfall is to interpret the robust maximum as an estimate with a guaranteed coverage level: the Wald interval is approximate, source uncertainty was held fixed, and the model set may omit the true target distribution.

In practice, report the assumption beside each transferred risk, check whether the fitted prevalence is feasible, and examine sensitivity to sampling and model uncertainty. Weighting can be unstable when target-to-source ratios are large; label-shift correction can be unstable when the class-conditional covariate distributions are nearly indistinguishable. Robustness is useful when its uncertainty set is explicit, but it cannot repair an implausible or incomplete set of assumptions.