Recognizing Reference Distribution Shift

Why Compare Query and Reference Distributions?

A cell-type classifier learns from a reference data set and is then applied to query cells. Even when the query cells receive plausible labels, their measured features may differ systematically from the reference. The difference may reflect tissue, biological condition, platform, preprocessing, cell-type composition, or some combination. Comparing distributions is a way to detect such mismatch and to ask whether probabilities learned in the reference are likely to remain informative in the query.

A distribution difference is a diagnostic signal, not a verdict. A query can have a different overall feature distribution simply because it contains different proportions of otherwise familiar cell types. Conversely, similar overall distributions do not guarantee that the relationship between features and labels is unchanged. Interpretation therefore depends on what is being compared and on how the query was collected and processed.

A feature may be a normalized expression value, a set of gene-expression measurements, or a representation constructed from them. A classifier may also produce a vector of label probabilities or scores. Comparing both features and scores can be useful: feature comparisons help identify where the inputs differ, while score comparisons show how the fitted model responds to the query. Neither comparison alone establishes that the model is wrong.

What Kind of Distribution Has Changed?

Three distinctions help organize the diagnosis. A change in the marginal feature distribution, \(P(X)\), means that query cells as a group have different feature values from reference cells. This is often called covariate shift. A change in the label proportions, \(P(Y)\), is a class-prior shift. A change in the feature–label relationship, \(P(Y\mid X)\), means that the same measured features no longer imply the same label probabilities. The last change is sometimes called conditional or concept shift.

These patterns can occur together. For example, a different tissue may contain new cell states, alter expression within familiar types, and change the proportions of types. A platform difference can shift measured expression even when the underlying biology is comparable. A condition can alter a marker gene within one cell type, changing how informative that gene is for classification. Observing a shifted \(P(X)\) does not, by itself, tell us which explanation applies.

The distinction matters for probabilities. If only \(P(X)\) changes while \(P(Y\mid X)\) stays the same, the reference conditional probabilities remain valid in principle, although a model may perform poorly in feature regions rarely seen during training. If \(P(Y\mid X)\) changes, reference probabilities need not describe the query. A difference in \(P(Y)\) can also change probabilities, even if the class-conditional feature distributions remain stable.

Comparing Features and Scores

Begin by ensuring that the comparison is meaningful. Query and reference features should use the same genes, scale, normalization, and model input definition. A difference caused by using unnormalized counts in one data set and normalized values in the other is not evidence of biological mismatch. Likewise, a model score comparison is interpretable only when the same fitted classifier and score definition are applied to both sets.

For individual features, inspect distribution plots and compare robust summaries such as medians and interquartile ranges, as well as means and standard deviations when those summaries are useful. A standardized mean difference can provide a scale-free descriptive comparison. If the reference and query means are \(\bar{x}_R\) and \(\bar{x}_Q\), and their standard deviations are \(s_R\) and \(s_Q\), one descriptive version is

The sign shows the direction of the difference, and the magnitude expresses it in units of the average within-group standard deviation. It is not a test statistic here, and there is no universal cutoff at which a shift becomes consequential. For high-dimensional features, examine a prespecified set of biologically or technically relevant features, or summaries of a representation used by the classifier; selecting only the largest observed differences can exaggerate their apparent importance.

For classifier outputs, compare the distributions of class probabilities, top scores, and possibly the margin between the largest and second-largest class scores. A shift toward lower top scores or smaller margins can indicate that query cells are less clearly supported by the reference model. It does not show that the model is calibrated in the query, and a high score is not proof of correctness. Score distributions should be read alongside feature comparisons and knowledge of the sampling design.

Separate Composition From Within-Type Differences

A whole-sample feature comparison can be driven by cell-type composition even when each type has the same feature distribution in both data sets. The overall mean is a weighted average of the type-specific means. If type \(k\) has proportion \(\pi_k\) and feature mean \(\mu_k\), the overall mean is \(\sum_k \pi_k\mu_k\). A change in \(\pi_k\) can therefore shift the overall mean while all the \(\mu_k\) values remain unchanged.

A useful diagnostic is therefore to compare overall distributions and, where there is adequate support, distributions within relevant cell groups or strata. If an overall shift largely disappears within groups, composition is a plausible contributor. If a shift remains within groups, it may reflect a condition effect, platform effect, unmodeled state variation, or a mismatch in the annotation. Grouping by predicted labels is not a fully independent check: prediction errors can hide or create apparent within-group patterns.

Why Shift Can Affect Label Probabilities

A classifier’s probability output is learned from the reference relationship between features and labels. To see why it may not transfer, consider the special case of class-prior shift: assume that the class-conditional feature distributions \(P(X\mid Y)\) are unchanged, but the label proportions differ. Bayes’ rule gives the query posterior as

where \(\pi_{R,k}=P_R(Y=k)\) and \(\pi_{Q,k}=P_Q(Y=k)\). The relationship follows by writing \(P_Q(Y=k\mid x)\propto P_Q(x\mid Y=k)P_Q(Y=k)\), substituting the unchanged class-conditional distribution, and using \(P_R(Y=k\mid x)\propto P_R(x\mid Y=k)P_R(Y=k)\). This adjustment requires the stated stability assumption and the query class proportions; a marginal feature shift alone does not justify applying it.

More generally, calibration in a reference population means that predicted probabilities agree with observed frequencies under that reference sampling distribution. It does not automatically guarantee the same agreement in a query population. A query shift may change which cells receive a given score or may move cells into feature regions with little reference support. Thus a distribution comparison informs how cautious to be about probability interpretation; it does not supply a universal correction.

Diagnostics, Comparisons, and Failure Modes

A formal two-sample test can supplement these checks, but a small p-value is not a measure of practical importance. With many cells, even a subtle difference may be detected; with correlated cells from a small number of donors, treating every cell as an independent replicate can overstate the evidence. Report the scale and direction of differences, and account for the sampling structure when making inferential claims. Descriptive plots and summaries are often more useful for locating a mismatch than a single significance test.

Univariate comparisons are easy to interpret but may miss shifts in combinations of features. Multivariate distances can summarize more dimensions, but their values depend on the representation, scaling, and distance definition. Neither approach diagnoses the biological cause on its own. A practical comparison uses complementary evidence: feature summaries, score distributions, known design information, and validation where labels are available.

Common failure modes include comparing incompatible preprocessing outputs, interpreting a composition-driven difference as within-cell expression change, and treating a low classifier score as proof of a novel cell type. The reverse error is also possible: apparently similar score distributions do not prove that query labels are reliable. A reference model can make confident predictions in a shifted setting. The central limitation is that unlabeled query distributions generally cannot reveal the true query relationship \(P(Y\mid X)\) by themselves.