Prior Probability Shift And Class-Conditional Invariance

Introduction

A classifier can face a new population even when the features associated with each class have not changed. For example, a screening system might encounter a larger proportion of patients with a disease than it did during development. If the distribution of test results within each disease class stays the same, this is prior probability shift: class prevalences change while class-conditional feature distributions remain invariant.

This assumption matters because it gives a direct way to express target risk using class-specific performance and target prevalences. It also has implications for posterior probabilities: a feature value that had one class probability in the source population can have a different probability in the target population. Prior probability shift is therefore distinct from covariate shift, which assumes that the conditional distribution of the label given the features stays fixed.

Definition and Risk Decomposition

Let \(P_s\) and \(P_t\) denote the source and target joint distributions of features \(X\) and a class label \(Y\). Suppose \(Y\) takes values in the finite set \(\{1,\ldots,K\}\). Write \(\pi_s(k)=P_s(Y=k)\) and \(\pi_t(k)=P_t(Y=k)\) for the source and target class prevalences. Assume \(\pi_s(k)>0\) for every class. This ensures that each source class-conditional distribution is defined. Target prevalences may be zero.

The invariant objects are the distributions of \(X\) within each class, not the overall feature distributions. Indeed, the feature marginal is a mixture of the class-conditional distributions, with mixing weights given by the class prevalences. Changing those weights will generally change the feature marginal even when every class-conditional distribution is unchanged.

Let \(h\) be a measurable prediction rule and let \(L(Y,h(X))\) be a measurable loss. Assume the loss is bounded, or more generally that the conditional expectations below are finite. Define the class-conditional risk \(R_k(h)=E_s[L(Y,h(X))\mid Y=k]\). Under prior probability shift this equals the corresponding target conditional expectation for every target-present class.

Proof. Partition the expectation defining target risk according to the value of \(Y\). Classes with zero target prevalence contribute zero. For each remaining class, the target conditional feature distribution equals the source conditional feature distribution, so its conditional expected loss is \(R_k(h)\). Therefore \(R_t(h)=\sum_k P_t(Y=k)E_t[L(Y,h(X))\mid Y=k]=\sum_k\pi_t(k)R_k(h)\). The same partition under the source distribution gives \(R_s(h)=\sum_k\pi_s(k)R_k(h)\). Subtracting these equalities proves the risk-change formula. \(\square\)

Because the prevalence differences sum to zero, the risk change depends on how class-specific risks differ, not on a uniform addition to every class risk. A classifier with unequal error rates across classes can have a different overall error rate solely because the class mixture has changed.

How Posterior Probabilities Change

Prior probability shift changes not only the average loss but also the posterior probability \(P(Y=k\mid X=x)\) used for probabilistic prediction or classification. To state the relationship precisely, suppose the class-conditional feature distributions have densities \(f_k\) with respect to a common dominating measure. Let \(f_s(x)\) and \(f_t(x)\) be the source and target feature-marginal densities. At points where the relevant marginal density is positive, Bayes’ rule gives \(P_s(Y=k\mid X=x)=\pi_s(k)f_k(x)/f_s(x)\) and \(P_t(Y=k\mid X=x)=\pi_t(k)f_k(x)/f_t(x)\).

Proof. The feature-marginal density is obtained by summing the joint density over the finite set of labels. Under the source distribution the joint density for class \(j\) is \(\pi_s(j)f_j(x)\), so summing gives \(f_s(x)=\sum_j\pi_s(j)f_j(x)\). The same argument with target prevalences gives \(f_t(x)=\sum_j\pi_t(j)f_j(x)\). Dividing the target joint density \(\pi_t(k)f_k(x)\) by the target marginal \(f_t(x)\) yields the displayed posterior. Dividing the posterior for \(a\) by the posterior for \(b\) cancels the common marginal denominator and gives the odds identity. \(\square\)

For two classes, the posterior odds under the target equal the source posterior odds multiplied by the change in prior odds: \(\{ \pi_t(1)/\pi_t(0)\}/\{\pi_s(1)/\pi_s(0)\}\). The class-conditional evidence \(f_1(x)/f_0(x)\) is unchanged; its balance against the prior odds is what changes. This is why a classifier’s source decision threshold need not remain appropriate in the target population.

Distinguishing Prior Probability Shift from Covariate Shift

The two shift assumptions hold different conditional distributions fixed. Under prior probability shift, \(P(X\mid Y)\) is invariant and \(P(Y)\) can change. Under covariate shift, \(P(Y\mid X)\) is invariant while the feature marginal \(P(X)\) changes. A change in the feature marginal alone does not identify which assumption is appropriate.

A density factorization makes the distinction explicit. Under prior probability shift, for a class with positive source prevalence the target-to-source joint density ratio is \(\pi_t(y)/\pi_s(y)\): it depends on the label, not on the feature value within that class. Under covariate shift, the conditional label distribution is shared, and the joint density ratio is determined by the feature-marginal ratio \(f_t(x)/f_s(x)\), wherever the source density is positive. These are different structural claims, not interchangeable explanations for a changed feature distribution.

Common Pitfall and Practical Meaning

Common mistake: “The target feature distribution differs from the source, so prior probability shift must hold; reweighting by the class prevalences will transfer risk.” This reasoning mistakes a marginal observation for a conditional invariance assumption. A changed feature marginal can result from covariate shift, prior probability shift, or other changes, and the example above shows that covariate shift can change both features and class prevalence while violating class-conditional invariance.

The correction is to keep the assumptions separate. Prior probability shift justifies the class-risk formula because each class-specific loss expectation transfers unchanged. If the conditional feature distribution within a class changes, then \(R_k(h)\) may change too, and prevalence reweighting alone does not generally recover target risk. Conversely, under covariate shift it is the label conditional on features that remains fixed; the class-conditional feature distribution need not.

This distinction is useful when interpreting performance changes. If class-specific performance is stable but the overall metric moves, changing prevalences may explain the movement under prior probability shift. But that explanation requires evidence or a defensible modeling assumption about class-conditional invariance; it cannot be concluded from a changed class mix or feature marginal alone. A further question is whether the target prevalences can be identified from available data, which requires additional analysis.