Marker Selection for Spot Deconvolution

Why Marker Selection Matters

A spatial spot contains expression from a mixture of cells, but deconvolution can distinguish cell types only through differences in their expression profiles. A gene expressed at a similar level by several cell types may be abundant and reproducible, yet provide little evidence about their proportions. Conversely, a highly distinctive gene may be too weak or variable in the reference to be dependable. Marker selection is therefore a balance: retain genes that are sufficiently expressed, distinguish the cell types of interest, and behave consistently in the reference data.

The target is not simply a list of familiar marker genes. It is a set of genes whose reference profiles allow the mixture proportions to be estimated under a stated measurement model. The introductory material on constrained regression for cell-type proportions establishes the role of a mixture signature and proportion constraints; here the focus is how the choice of genes changes the information in that signature.

This expression-scale model is an approximation. It is appropriate when the reference and spot values have been put on compatible scales and mixing is reasonably represented by a weighted average. For raw counts, exposure or library size must also be accounted for; a count likelihood is a different observation model, not a reason to treat raw counts as directly comparable expression levels. In either setting, the reference profiles must correspond closely enough to the cells in the tissue. Differences between reference and tissue expression can bias deconvolution even when a gene appears distinctive in the reference.

Marker selection also depends on the question. A gene that separates one pair of cell types may not separate another pair, and a panel for estimating broad cell classes need not be suitable for distinguishing closely related subtypes. Candidate markers should therefore be assessed against the full set of cell types being estimated, not only against a chosen target type.

Three Criteria for Candidate Markers

Specificity describes how much more a gene is expressed in one cell type than in competing types. For two types, a simple contrast for a gene intended to identify type A is \(r_{gA}-r_{gB}\). A relative contrast can be useful when genes differ greatly in overall abundance. For more than two types, compare the target profile with the strongest competing profile, while also checking whether the gene distinguishes the other types that must be estimated.

Expression adequacy asks whether the gene is detected often and strongly enough to be useful in the spot data. A large reference contrast based on near-zero values can be fragile: sampling zeros, low counts, or normalization noise may overwhelm it. Expression adequacy is not the same as specificity. A common, highly expressed gene may be measurable but shared, while a specific gene may be too rare for reliable measurement.

Reference stability concerns whether the gene's cell-type profile is consistent across reference donors, batches, or other relevant strata. Stability should be assessed for the contrast between cell types, not only for each mean in isolation. A marker can have a stable average level in a target type but an unstable difference from its competitors. Reference stability does not guarantee that the gene is stable in the spatial tissue; it is evidence about the reliability of the reference signature.

A practical screen begins with reference pseudobulk profiles by cell type and, where possible, by donor. For each candidate, inspect the mean expression, the between-type contrast, detection frequency, and variability across reference samples. A contrast-to-total score can summarize specificity for an initial ranking, but it is a screening aid rather than a universal selection rule. For example, for two types one possible score is

where \(\epsilon>0\) is a small fixed offset in the same expression units, chosen to avoid an unstable ratio when both profiles are near zero. This score does not incorporate spot detection or uncertainty in the reference, so it should not be used alone.

Why Shared Markers Can Destabilize Proportions

The key issue is not whether a gene is called a marker in a biological description, but whether its reference expression differs in a way that identifies the mixture. Consider two cell types, A and B, and let \(p\) be the proportion of A, so the proportion of B is \(1-p\). For each selected gene,

If the errors are independent with mean zero and common variance \(\sigma^2\), and the reference profiles are treated as fixed and correct, least squares estimates \(p\) by minimizing the sum of squared residuals. Differentiating that sum with respect to \(p\) gives the following unconstrained estimate:

Because \(p\) is a proportion, the fitted value may additionally be constrained to the interval from 0 to 1. The formula shows what carries information about \(p\): the squared contrasts \(d_g^2\). A shared gene with \(d_g=0\) contributes no information about the A-versus-B proportion. If all selected genes are shared, the denominator is zero and the proportion is not identifiable from this panel.

The variance follows directly from the same expression. Substituting \(y_g-r_{gB}=p d_g+\epsilon_g\) into the estimate yields \(\hat p-p=\sum_g d_g\epsilon_g/\sum_g d_g^2\). Independence and common error variance then give

Thus, when the panel's useful contrasts are small, noise is amplified. In a multi-type model the corresponding issue is poor separation among the columns of the selected reference matrix: if several cell types have nearly identical profiles over the selected genes, distinct proportion combinations can produce nearly the same fitted expression. Reference error or systematic expression shifts can make this worse by moving the estimated mixture in directions that the panel can barely distinguish.

Assembling and Checking a Panel

Selection is best treated as an iterative design decision, not as a single thresholding rule. Begin with candidates that have plausible cell-type contrasts, then check that their expression and detection are adequate in the spatial assay and that their reference contrasts persist across donors or batches. Examine the selected reference matrix as a whole: a collection of individually plausible genes can still fail if their profiles do not distinguish all cell types jointly. Repeatedly selecting genes from the same correlated biological program can also make apparent evidence less diverse than the gene count suggests.

The assumptions behind the calculations should be checked before interpreting the fitted proportions. At a minimum, the reference and spot measurements should be on compatible scales; the selected genes should have sufficiently reliable reference profiles; the mixture should be reasonably approximated by the chosen linear or count model; and the residual errors should not be dominated by unmodeled gene-specific shifts. For the displayed variance formula, errors must additionally be independent, mean zero and equally variable, with fixed reference profiles. If errors differ in variance, weighted fitting is more appropriate; if errors are correlated, the covariance structure matters.

Useful diagnostics include plots of reference expression by cell type and donor, spot detection rates for selected genes, residuals by gene, and sensitivity of estimated proportions to removing or replacing individual markers. Large changes after dropping one gene indicate that the panel may be relying excessively on that gene. Residual patterns concentrated in one cell type or region can signal reference mismatch or a model failure rather than a need to add more markers. The earlier material on reference mismatch in spatial deconvolution is relevant when those patterns suggest that the reference profile itself does not represent the tissue.

This strategy differs from selecting genes solely by differential expression. A differential-expression test asks whether a group contrast is supported under its model; marker selection asks whether a gene helps identify mixture proportions in the target assay. A small p-value does not ensure adequate spot detection, stable reference contrasts, or good separation from every other cell type. Conversely, a marker panel should not be selected merely to maximize a training-set fit, which can favor reference-specific noise.

Alternative approaches include using a larger, carefully screened gene set, weighting genes by measurement reliability, or using a model that accounts for count variability and reference uncertainty. These approaches can be preferable when their assumptions match the data, but none can recover proportions that are unidentifiable because the cell types have indistinguishable profiles. Marker selection is also limited by biology: cell states can vary within a nominal type, and genes can change with tissue context. A panel that performs well in one reference or tissue may not transfer unchanged to another.