Benchmarking Deconvolution Without Ground Truth

Assessing Deconvolution When Proportions Are Unknown

Spatial deconvolution estimates which cell types contribute to each spatial spot. In real tissue, the true proportions for each spot are usually unavailable, so a direct comparison of estimated and true proportions is impossible. This makes evaluation a collection of complementary checks rather than a single accuracy score. The checks considered here ask whether the method behaves plausibly under controlled reference changes, agrees with independent marker evidence, leaves interpretable residuals, and is robust to reasonable choices of reference data.

These checks do not turn unknown proportions into known ones. A method can fit expression well and still estimate the wrong mixture if cell types have similar signatures, the reference is mismatched, or important cell states are absent. The introductory material on evaluating deconvolution with synthetic mixtures covers cases where mixtures with known construction proportions can be made. Here, the target is the harder setting in which the observed tissue has no such ground truth.

What Is Being Checked?

Let \(y_{sg}\) denote normalized expression of gene \(g\) in spot \(s\), and let \(A_{kg}\) denote the reference expression signature for cell type \(k\) and gene \(g\), on a compatible scale. A simple mixture representation is

Here \(p_{sk}\) is the estimated contribution of cell type \(k\) to spot \(s\). Depending on how the signatures and normalization are defined, this contribution may represent an RNA contribution rather than a literal fraction of cells. The fitting procedure produces \(\hat p_{sk}\) and fitted expression \(\hat y_{sg}=\sum_k \hat p_{sk}A_{kg}\). The residual \(e_{sg}=y_{sg}-\hat y_{sg}\) records the expression not accounted for by that fitted mixture.

For any benchmark to be interpretable, the observed expression and reference signatures must be put on compatible scales, and the same normalization and fitting procedure should be used across comparisons. If evaluation genes helped determine the fitted proportions, agreement on those genes is not independent evidence: the method was explicitly given the information being checked. A practical approach is to define fitting genes and validation genes in advance, then leave the validation genes out of the fit.

Four questions organize the evaluation:

• Does the pipeline respond sensibly to a controlled change? Hold out reference donors or cell types and examine whether expected failures become visible.
• Do independently measured markers agree with the estimates? Check genes that were not used to fit the proportions.
• What remains unexplained? Look at residual size, gene patterns, and spatial organization.
• Are conclusions sensitive to plausible reference choices? Compare estimates from references that differ in donor, condition, or processing.

Reference Holdouts: Testing Expected Behavior

A reference holdout is a controlled stress test. One option is to exclude cells from a donor or batch when building the reference, then assess whether the fitted signatures and spot estimates remain plausible when that held-out material is evaluated. A more severe test is to omit one cell type from the reference. If spots contain that type, the model cannot assign its contribution to the missing type. Instead, expression from the omitted type may be left in the residuals or incorrectly attributed to similar represented types.

Holdouts are most informative when their design reflects a specific concern. Holding out donors tests dependence on donor-specific signatures; holding out a cell type tests what happens when the reference is incomplete. In either case, keep the test data separate from reference construction and from choices made during tuning. If the held-out cells were used to select markers or optimize the method, the test is no longer an independent check.

Independent Marker Consistency

Marker consistency asks whether estimated contributions vary in a direction that agrees with expression of cell-type markers not used for fitting. For a cell type with validation-marker set \(M\), one simple score for spot \(s\) is the mean normalized expression across those genes:

The score can be compared with \(\hat p_{sk}\), the estimated contribution of the corresponding cell type. Agreement might appear as a positive association across spots or as higher marker scores in spots with larger estimated contributions. The expected pattern should be specified with biological and technical context in mind: marker genes may vary with activation, tissue region, sequencing depth, or related cell types. Markers are supporting evidence, not a direct measurement of cell abundance.

Residual Structure: What the Fit Leaves Behind

A good global fit can conceal systematic local failures. Examine residuals by gene, spot, tissue region, and spatial neighborhood. A residual that is positive for a coherent set of markers may point to an omitted cell type or state. Residuals concentrated in one region may indicate spatially varying reference mismatch. A broad increase in residual magnitude in low-depth spots may instead reflect technical variation or an unsuitable expression scale.

The justification for this diagnostic follows directly from the mixture model. Suppose the true mean expression includes a cell type \(h\) omitted from the reference, with contribution \(p_{sh}\) and signature \(A_{hg}\). If the fitted model accounts for the represented types but not this omitted component, then, in a simplified case,

where \(\varepsilon_{sg}\) is expression noise and other modeling error. Thus residuals tend to be positive for genes strongly expressed by the omitted type, especially in spots where its contribution is large. The implication is conditional, not conclusive: a mismatch in gene expression or normalization can also generate residuals with similar patterns.

Spatial patterning can be summarized descriptively before applying a formal spatial test. For a chosen gene, compare residuals at neighboring spots. If nearby spots repeatedly have residuals of the same sign, the fit may be missing spatially organized biology or may have a region-specific mismatch. A spatial test is not automatically a test of deconvolution accuracy; it tests whether residual organization is inconsistent with the chosen model's assumptions.

Sensitivity to Reference Choice

A deconvolution result may depend on which reference cells, donors, or conditions are used. Refit the same spots under several defensible reference choices while holding the gene set, normalization, and fitting settings fixed as far as possible. Then compare estimates for each cell type and spot. For two references \(A\) and \(B\), a simple summary for type \(k\) across \(S\) spots is the mean absolute difference

This quantity measures disagreement between the two runs in the units of the estimated contribution. Small disagreement suggests stability to that particular reference change; it does not imply accuracy. Large disagreement flags dependence on the reference and motivates investigation of signature similarity, marker coverage, and whether either reference better represents the tissue. Compare not only averages but also spatial maps and which spots change most.

Combining Evidence and Recognizing Failure Modes

The strongest evaluation uses checks that fail for different reasons. A reference holdout probes the pipeline under a planned perturbation; held-out markers assess expression consistency; residual analysis searches for unexplained structure; and reference sensitivity measures dependence on reference choice. Agreement across these checks increases confidence that the output is not obviously contradicted by available evidence. It is still not equivalent to validation against true proportions.

This differs from benchmarking only with synthetic mixtures, where the constructed proportions are known and numerical errors against those proportions can be calculated. Synthetic mixtures can directly test recovery under their construction assumptions, but they may not reproduce tissue complexity. Ground-truth-free checks apply to real tissue, but their conclusions are indirect. They complement rather than replace one another.

Common pitfalls include:

• Calling a low residual error accurate deconvolution. Several distinct mixtures can produce similar expression. A good fit shows compatibility with the model, not unique recovery of the true proportions.
• Reusing fitting markers as validation markers. Agreement is partly guaranteed when the same genes helped determine the estimates. Reserve independent genes for the check.
• Treating a holdout as a direct estimate of real-world accuracy. A holdout tests a particular change, such as omitting a donor or cell type; its result does not cover every source of mismatch.
• Equating reference stability with truth. Multiple references can share the same bias. Conversely, a real biological difference between reference sources can produce sensitivity without identifying which estimate is wrong.
• Ignoring technical and biological alternatives. Marker variation and structured residuals may reflect activation, tissue region, normalization, or measurement quality rather than an incorrect cell-type contribution.

When diagnostics disagree, report the disagreement rather than selecting whichever check is most reassuring. State which genes and references were held out, how markers were chosen, which residual patterns were examined, and which choices materially changed the estimates. Such reporting makes the evidence interpretable while keeping the conclusion within its limits.