Propagating Label-Shift Uncertainty Into Risk

Why Uncertainty Must Be Propagated

Under label shift, the class proportions change between source and target populations, while the distribution of features conditional on the class remains the same. A target class prevalence is usually not observed directly: it is inferred from target predictions and a confusion matrix estimated using labeled source data. Both ingredients are uncertain. Treating the estimated confusion matrix as exact can therefore make a target-risk interval or decision threshold look more precise than the data support.

The goal here is to carry uncertainty through the whole calculation. We first construct a confidence set for the target class-prevalence vector. We then use that set to bound target risk and, in the binary case, the source-posterior threshold corresponding to a target cost-sensitive decision rule.

A Confidence Set for Target Prevalences

Let the class label take values in \(\{1,\ldots,K\}\). Write \(\pi_j=P_t(Y=j)\) for the target prevalence of class \(j\), and let \(\pi=(\pi_1,\ldots,\pi_K)\). The vector \(\pi\) belongs to the probability simplex: its entries are nonnegative and sum to one. Let \(\hat Y\) be the prediction of a fixed classifier, and define the class-conditional confusion matrix by

Thus column \(j\) describes the predicted-class distribution for true class \(j\); every column sums to one. Under label shift, \(P_t(\hat Y=i\mid Y=j)=A_{ij}\). Consequently, if \(q_i=P_t(\hat Y=i)\), then \(q=A\pi\).

Suppose a labeled source validation sample gives estimates \(\hat A_{ij}\), and \(m\) target observations give predicted-class proportions \(\hat q_i\). For a transparent finite-sample construction, assume the validation observations are independent within each true class, and the target observations are independent draws from the target population. Assume also that the classifier is fixed independently of these validation observations, or that the validation data are otherwise valid for estimating its confusion matrix. The following Hoeffding bounds give one possible simultaneous uncertainty set; other valid simultaneous confidence bounds can be substituted.

Let \(n_j\) be the number of validation observations in class \(j\), and choose a total error probability \(\alpha\). Set

These radii use a union bound over the \(K^2\) confusion-matrix entries and the \(K\) target prediction proportions. Define the prevalence confidence set

This set is determined by linear inequalities after using the two sides of each absolute-value constraint separately. It is therefore a convex polytope, and extrema of a linear function over it can be found by linear programming. If an estimated column has no observations, the displayed Hoeffding radius is not defined; that class requires a different valid bound or additional validation data.

Worked Examples of Prevalence Uncertainty

Why the Prevalence Set Covers the Truth

The coverage claim follows directly from simultaneous bounds on the two sampled quantities. The proof also explains why the matrix-error term is weighted by the unknown prevalences rather than simply discarded.

The true \(\pi\) is a probability vector, so it also satisfies the simplex constraints. It therefore belongs to \(\mathcal C_\pi\) on the simultaneous event, whose probability is at least \(1-\alpha\). \(\square\)

Propagating Prevalence Uncertainty into Target Risk

Let \(L\) be a loss with \(0\leq L\leq1\), and define the class-conditional risk \(r_j=E_s[L\mid Y=j]\). Under label shift, the conditional distribution of the features given the class is unchanged, so the same \(r_j\) applies in the target population. The target risk is consequently

If the class-conditional risks are known, minimize and maximize this linear function over \(\mathcal C_\pi\). If they are estimated, include their uncertainty too. For example, suppose simultaneous intervals \(\underline r_j\leq r_j\leq\overline r_j\) hold on an event with the desired coverage. Since \(\pi_j\geq0\), a valid risk interval on the intersection of the prevalence and risk events is

The endpoints are conservative if the estimated risks and prevalences are statistically dependent, but they remain valid when all component intervals hold simultaneously. A union bound can allocate the total error probability across the prevalence and risk intervals.

Propagating Prevalence into a Decision Threshold

For a binary problem, let \(p=P_t(Y=1)\) and let \(\eta_s(x)=P_s(Y=1\mid X=x)\) be a calibrated source posterior. Under label shift, Bayes' rule gives the target posterior odds by multiplying source posterior odds by the target-to-source prevalence odds ratio. If the source prevalence is \(p_s\), then

Suppose predicting class 1 incurs false-positive cost \(c_{\mathrm{FP}}>0\), while predicting class 0 incurs false-negative cost \(c_{\mathrm{FN}}>0\). The target Bayes rule predicts class 1 when \(\eta_t(x)\geq\tau\), where \(\tau=c_{\mathrm{FP}}/(c_{\mathrm{FP}}+c_{\mathrm{FN}})\): the two conditional expected costs are \(c_{\mathrm{FP}}(1-\eta_t)\) and \(c_{\mathrm{FN}}\eta_t\). Solving the odds equation for the source posterior gives the equivalent source-score cutoff

For fixed source prevalence and costs, \(a(p)\), and hence \(t_s(p)\), decreases as \(p\) increases. Therefore a prevalence interval \([\underline p,\overline p]\) yields the source-cutoff interval \([t_s(\overline p),t_s(\underline p)]\). This is an uncertainty interval for the appropriate threshold, not a guarantee that the score itself is calibrated or that label shift holds.

Common Pitfall and Practical Interpretation

A frequent mistake is to plug \(\hat A\) and \(\hat q\) into \(q=A\pi\), solve for one prevalence estimate, and then treat that estimate as known. The correction is to retain the sampling uncertainty in both quantities, form a set of prevalence vectors compatible with those errors, and optimize the risk or threshold over that set. For multiclass problems, the prevalence set may be wide or even empty under a chosen set of bounds; an empty set signals incompatibility between the data and constraints, not evidence that the true prevalence is absent.

These guarantees are conditional on the label-shift model and on valid sampling and confidence bounds. They do not protect against changes in the class-conditional distributions. A narrow interval can therefore reflect precise estimation under the model without establishing that the model itself is appropriate.