Diagnosing Conditional Shift in Classification

Why Diagnose the Shift Mechanism?

A classifier can perform differently after deployment for more than one reason. The class proportions may have changed while the feature distribution within each class stayed stable. The feature distribution may have changed while the relationship between features and labels stayed stable. Or the relationship itself may have changed. These mechanisms imply different corrections: a prior adjustment or importance weighting is justified only under its corresponding assumptions.

A labeled target audit sample—target observations for which the true class is recorded—allows direct checks of those assumptions. The central comparison is not merely whether the overall error rate changed. It is whether changes in features and errors within classes or meaningful feature regions agree with the proposed shift model.

These assumptions are not interchangeable. Under prior shift, class-conditional features are stable, but posteriors generally change because class priors changed. Under covariate shift, posteriors are stable, but class-conditional feature distributions can change as a consequence of the altered feature marginal. Thus, observing that \(P_t(X\mid Y)\) differs from \(P_s(X\mid Y)\) rules out prior shift, but does not by itself rule out covariate shift.

What Prior Shift Predicts for Posteriors

For a class \(k\), write \(\pi_{s,k}=P_s(Y=k)\) and \(\pi_{t,k}=P_t(Y=k)\). Under prior shift, Bayes' rule gives a precise prediction for the target posterior. For any \(x\) where the relevant conditional densities or mass functions are defined and the denominator is positive,

assuming \(\pi_{s,k}>0\) for every class whose target prior is positive. In words, prior adjustment multiplies each source posterior by the target-to-source prior ratio and then renormalizes. This is a prediction that can be checked against representative labeled target data; a changed posterior alone is not evidence against prior shift, because prior shift itself changes posteriors.

Proof. Let \(f_k(x)\) be a density or mass function for \(X\mid Y=k\), common to source and target by assumption. Bayes' rule under the target distribution gives

Under the source distribution, \(P_s(Y=k\mid X=x)=\pi_{s,k}f_k(x)/\sum_j\pi_{s,j}f_j(x)\). Multiplying this source posterior by \(\pi_{t,k}/\pi_{s,k}\) gives \(\pi_{t,k}f_k(x)/\sum_j\pi_{s,j}f_j(x)\). The common denominator across classes cancels when these adjusted values are normalized, leaving the target formula. \(\square\)

Diagnosing the Assumptions With Labeled Audits

A useful audit separates checks by the assumption being tested. To assess prior shift, compare feature distributions within each class across source and target. For a feature region \(B\), compare \(P_s(X\in B\mid Y=k)\) with \(P_t(X\in B\mid Y=k)\). Differences beyond sampling variation are evidence against prior shift. The audit should also compare target posterior behavior with the prior-adjusted source predictions.

To assess covariate shift, compare conditional label behavior across populations at comparable feature values or regions. For example, within a region \(B\), compare \(P_s(Y=k\mid X\in B)\) with \(P_t(Y=k\mid X\in B)\). Because conditioning only on a coarse region can hide variation within that region, this is a diagnostic approximation rather than a universal test of equality at every \(x\). Calibration and proper-score diagnostics from earlier material can summarize prediction behavior, but an aggregate score alone cannot establish which shift mechanism holds.

Classification errors provide a complementary diagnostic. For a fixed rule \(h\), define its class-specific error as \(P(h(X)\ne Y\mid Y=k)\), and examine it in scientifically meaningful feature regions as well as overall. A changed error rate within a class or region may reveal that the classifier encounters different cases or that its conditional predictions no longer match the labels. Error patterns are evidence, not a unique identification of mechanism: a changed mixture of subgroups can alter aggregate error even when error behavior within each subgroup is stable.

A Direct Within-Class Comparison

For a binary feature or a prespecified feature region, an audit can estimate class-conditional probabilities separately in source and target. If source and target samples are independent random samples within class \(k\), a two-proportion comparison quantifies whether an observed difference is larger than sampling variability. A class-stratified audit is especially useful for this purpose: it gives adequate observations in each class, though it does not reproduce the target population class proportions.

The following calculation illustrates evidence against prior shift. It does not establish that every aspect of the conditional distribution has changed; it tests the selected feature-region comparison.

Why Subgroup Error Patterns Matter

An overall error rate can conceal compensating changes. If errors rise in one feature region and fall in another, the aggregate may remain nearly constant. Conversely, the aggregate can change merely because the target contains different proportions of regions with different fixed error rates. A labeled audit therefore benefits from reporting error counts and rates by class and prespecified, relevant feature regions, alongside the aggregate.

A useful formal identity clarifies this point. Let \(G=g(X)\) be a fixed, measurable subgroup label and \(h\) a fixed classifier. When the probabilities are defined,

This follows by partitioning the error event over the disjoint events \(\{G=g,Y=k\}\). Thus aggregate risk depends both on subgroup/class frequencies and on conditional error rates. A changed aggregate does not identify which factor changed. In a class-stratified target audit, the sampling fractions across classes may have been chosen by design, so raw pooled error is not automatically the target-population risk; use appropriate target class proportions or design weights for that estimand.

Limits and Common Pitfalls

No finite audit can verify equality of entire conditional distributions from a few comparisons. The diagnostic depends on which features, regions, and posterior summaries are examined. Small target class counts give imprecise estimates; many unplanned comparisons inflate the chance of finding an apparent discrepancy. Prespecifying important comparisons, reporting uncertainty, and treating exploratory findings as exploratory help prevent overclaiming.

Support is also essential. If a target feature region has no source observations, source conditional behavior there cannot be checked or reliably transferred. Likewise, if the audit selectively samples cases in a way that depends on features within class, naive estimates of \(P_t(X\mid Y)\) may be biased unless the sampling design is accounted for. A diagnostic that appears to support an invariance assumption in an overlap region says nothing automatic about areas without overlap.

• Mistake: calling every posterior change evidence against prior shift. Prior shift changes posteriors when priors change. Compare observed target posteriors with the prior-adjusted source predictions.
• Mistake: treating a change in \(P(X\mid Y)\) as proof against covariate shift. Covariate shift requires stable \(P(Y\mid X)\), not stable class-conditional feature distributions.
• Mistake: trusting an unchanged overall error rate. Different subgroup changes can cancel. Inspect conditional patterns and uncertainty, not only the aggregate.
• Mistake: interpreting a noisy audit discrepancy as a mechanism. Sampling variability, multiplicity, selection, and limited overlap can all produce misleading comparisons.