Identifiability Of Label Shift From Confusion Matrices

Why Confusion-Matrix Identifiability Matters

Under label shift, the distribution of features within each class is unchanged between source and target populations, while the class prevalences may change. This makes it possible, in principle, to use a fixed classifier’s predictions on target observations to learn about target class prevalences. But that is possible only if the classifier’s class-conditional prediction patterns distinguish the classes. The relevant question is therefore not merely whether the classifier makes errors; it is whether its confusion probabilities preserve enough information to separate the target class proportions.

We make that question precise using a finite number \(K\) of classes. The classifier is fixed throughout, and its predictions are evaluated under the source population to form a population confusion matrix. The key result is an exact criterion: target class prevalences are globally identifiable from target predicted-label frequencies if and only if that matrix has full column rank.

The Linear System Under Label Shift

Each column of \(C\) is a probability distribution over predicted classes. In particular, its entries are nonnegative and sum to one: \(\sum_{i=1}^K C_{ij}=1\). The prevalence vector also lies in the probability simplex \(\Delta^{K-1}=\{u\in\mathbb{R}^K:u_j\geq 0,\ \sum_j u_j=1\}\).

Label shift means class-conditional invariance: \(P_t(X\mid Y=j)=P_s(X\mid Y=j)\) for each class \(j\) whose target prevalence is positive. Since the classifier \(h\) is the same in both populations, this implies \(P_t(h(X)=i\mid Y=j)=C_{ij}\). Applying the law of total probability gives the system linking predicted frequencies to prevalences.

The equation is a population statement: it relates exact probabilities, not sample proportions. It uses class-conditional invariance and a common fixed classifier. If the classifier changes, or if its prediction probabilities within a class change across populations, the same source matrix need not link \(q_t\) and \(\pi_t\).

Identifiability and the Rank Criterion

This definition is about uniqueness, not about how to compute an estimate from finite data. The set of feasible predicted-frequency vectors is \(\{Cu:u\in\Delta^{K-1}\}\), the convex hull of the columns of \(C\). If two columns or combinations of columns overlap in a way that gives distinct simplex vectors the same image, the observed predicted frequencies cannot distinguish those prevalences.

Proof. First suppose \(C\) has full column rank. Then its null space contains only the zero vector. If \(u,v\in\Delta^{K-1}\) satisfy \(Cu=Cv\), then \(C(u-v)=0\). Full column rank implies \(u-v=0\), so \(u=v\). Thus the map is one-to-one on the simplex.

Conversely, suppose \(C\) does not have full column rank. There is a nonzero vector \(a\in\mathbb{R}^K\) such that \(Ca=0\). Let \(\mathbf{1}\) denote the vector of \(K\) ones. Column-stochasticity gives \(\mathbf{1}^\mathsf{T}C=\mathbf{1}^\mathsf{T}\), and hence \(\mathbf{1}^\mathsf{T}a=\mathbf{1}^\mathsf{T}Ca=0\). Thus the entries of \(a\) sum to zero.

Consider the interior simplex point \(u_0=(1/K,\ldots,1/K)^\mathsf{T}\). Because \(a\neq 0\), we can choose \(\epsilon>0\) small enough that every entry of both \(u_0+\epsilon a\) and \(u_0-\epsilon a\) is nonnegative. Both vectors sum to one because \(\mathbf{1}^\mathsf{T}a=0\), so both belong to the simplex. They are distinct, and \(C(u_0+\epsilon a)=Cu_0=C(u_0-\epsilon a)\). The map is therefore not one-to-one. This proves both directions. \(\square\)

The result is global over the full simplex. At a particular boundary prevalence vector, a rank-deficient matrix can still yield a unique feasible solution; the theorem does not claim otherwise. It says that without restricting the possible prevalences in advance, rank deficiency prevents global identification.

What Full Rank Does—and Does Not—Guarantee

Full column rank guarantees uniqueness in the exact population system when the model is compatible, meaning the observed \(q_t\) lies in the image of the simplex under \(C\). It does not guarantee that the unique prevalence vector can be recovered precisely from noisy estimates of predicted frequencies or an estimated confusion matrix. A matrix can be full rank but nearly rank deficient: its columns are technically independent but very close to lying in a lower-dimensional subspace.

For a square invertible \(C\), a perturbation \(\delta q\) in predicted frequencies leads to \(\delta\pi=C^{-1}\delta q\), and therefore

If \(\|C^{-1}\|\) is large, small frequency errors can produce large changes in the recovered prevalences. For a rectangular matrix with full column rank, the analogous statement uses a left inverse. Thus identifiability is a uniqueness property; stability concerns sensitivity to perturbations. Rank alone answers the first question, not the second.

Common Mistake

• Wrong reasoning: “The classifier has nonzero accuracy for every class, so target prevalences are identifiable.” Nonzero diagonal entries do not rule out linear dependence among the columns.
• Correction: Check whether the full columns of the population confusion matrix are linearly independent. Even a useful classifier can have identical or dependent class-conditional prediction distributions, in which case some distinct prevalences produce exactly the same predicted-label frequencies.
• Another distinction: A unique solution to the population equation does not mean sample-based recovery is stable. Near-dependence can make an otherwise identifiable system highly sensitive to estimation error.