Diagnosing Shift Mechanisms From Source And Target Samples

What Can a Shift Diagnostic Establish?

A source sample with labels and a target sample without labels provide different kinds of evidence. The source data reveal the joint relationship between covariates and outcomes in the source population. The target covariates reveal how the input distribution may have changed. A fixed prediction rule applied to those covariates gives a further observable summary, but it does not reveal the target outcomes themselves.

The central task is therefore not to name a shift mechanism from one statistic. It is to ask which observable consequences follow from each mechanism, check those consequences against the available samples, and keep distinct mechanisms that the observations cannot distinguish.

These definitions describe population distributions, not sample patterns. In practice, a diagnostic can support or contradict implications of a shift assumption; it rarely proves the assumption from finite data. In particular, finding changed covariate distributions is evidence against identical source and target covariates, but it does not decide whether the conditional distribution of \(Y\) given \(X\) stayed fixed.

Comparing Source and Target Covariates

A two-sample diagnostic asks whether the observed source covariates could plausibly have come from the same distribution as the target covariates. One flexible approach is to combine the samples, label each observation by its domain (source or target), and fit a classifier to predict that domain. Evaluation must use held-out observations: a classifier that memorizes its training data does not provide a useful comparison.

For an ideal domain classifier, performance has a precise population interpretation. Let \(p\) and \(q\) be densities of \(P_S^X\) and \(P_T^X\) with respect to a common dominating measure. Suppose source and target are equally likely before seeing \(X\). The classifier predicts the source for covariates with the larger source density and the target otherwise. Its accuracy is related to the total variation distance, defined here as \(\operatorname{TV}(P_S^X,P_T^X)=\frac12\int |p-q|\).

To verify the formula, at each covariate value the best decision selects the domain with the larger density. The probability of a correct decision is therefore:

The first equality uses the equal domain probabilities. The second follows from \(\max\{a,b\}=(a+b+|a-b|)/2\) and the fact that each density integrates to one. This result concerns the best possible classifier. A practical classifier may perform near chance because it is poorly specified, underfit, or trained on too few observations, even when the distributions differ. Conversely, estimated performance needs uncertainty assessment and a held-out evaluation before it is interpreted as evidence.

Other useful comparisons include prespecified summaries, feature-wise distributions, and two-sample tests. A domain classifier can detect differences involving combinations of covariates that individual summaries miss, but its result depends on the classifier class and the available sample size. A weak domain classifier is not proof of equality, and a detected difference is not itself proof of covariate shift: it only concerns \(P_S^X\) versus \(P_T^X\).

Using Target Predictions to Check Label Shift

Label shift has an observable implication when a prediction rule is fixed and applied to both domains. Let \(g(X)\) be a classifier with classes \(1,\ldots,K\), and define its source confusion probabilities by \(C_{a y}=P_S(g(X)=a\mid Y=y)\). This convention places predicted classes in rows and true classes in columns, so each column sums to one. If class-conditional distributions are unchanged, the same probabilities \(C_{a y}\) apply in the target population.

Let \(\pi_T\) be the vector of target class probabilities, with entries \(\pi_{T,y}=P_T(Y=y)\). Let \(q_T\) be the vector of target predicted-class proportions, with entries \(q_{T,a}=P_T(g(X)=a)\). Under label shift, conditioning on the true class gives:

This is a diagnostic relationship, not an automatic confirmation of label shift. The source labels can be used to estimate \(C\); target covariates can be passed through the same fixed \(g\) to estimate \(q_T\). If the relationship cannot be reconciled with a plausible target class-probability vector, label shift is contradicted for the assumed stable confusion probabilities. If it fits, label shift remains plausible, but other changes can produce the same predicted proportions.

The diagnostic depends on the classifier being the same fixed function in both domains and on the source confusion probabilities being informative about the target. If the classifier is retrained using target data, the comparison no longer uses the same \(g\). If \(C\) is nearly singular, substantially different class proportions can produce similar prediction proportions, so the inferred proportions are unstable. Hard predictions also discard information: score-based summaries can be useful, but their own class-conditional stability must be justified rather than presumed.

Why Unlabeled Data Cannot Rule Out Conditional Shift

The covariate comparison and the prediction-proportion check use only observable quantities: source \((X,Y)\), target \(X\), and predictions computed from \(X\). None directly observes target \(Y\). This creates a fundamental limit, not merely a weakness of a particular test.

Proof. Fix any target covariate distribution \(Q_X\). Choose two different conditional distributions \(K_1(Y\mid X)\) and \(K_2(Y\mid X)\). Define target joint distributions by \(Q_1(dx,dy)=Q_X(dx)K_1(dy\mid x)\) and \(Q_2(dx,dy)=Q_X(dx)K_2(dy\mid x)\). Both produce exactly the same target covariate marginal \(Q_X\). The source distribution is also identical under the two constructions. Thus the distribution of all observed data—labeled source observations and unlabeled target covariates—is the same, even though the target conditionals differ. Any diagnostic based only on those observations has the same sampling distribution in both cases, so it cannot generally distinguish them. \(\square\)

Interpreting the Diagnostics Together

A disciplined interpretation separates observable findings from assumptions about outcomes. The following table summarizes what the checks can and cannot say.

Use uncertainty estimates for all sample-based comparisons. For the domain classifier, choose evaluation data not used to fit it and report uncertainty in its held-out performance. For the label-shift equation, account for uncertainty in both the estimated source confusion matrix and the target prediction proportions. Also examine whether conclusions are sensitive to the chosen representation and prediction rule. A result based on one classifier is a result about that classifier’s summaries, not a complete comparison of distributions.

Common Mistake

Wrong reasoning: “The source and target covariate distributions look similar, and the target prediction proportions match the label-shift equation; therefore the target conditional distribution must be unchanged.”

Correction: Both checks concern observable covariates or predictions. Different target conditionals can produce the same covariate distribution and prediction proportions, as the discrete example shows. These checks can expose contradictions and support a shift assumption, but unlabeled target data alone cannot generally verify \(P_T(Y\mid X)=P_S(Y\mid X)\). That requires additional assumptions, target labels, or other information that links target outcomes to observed quantities.