Evaluating Deconvolution With Synthetic Mixtures

Why Evaluate With Synthetic Mixtures?

Deconvolution estimates the proportions of cell types contributing to each spatial spot. In a tissue section, the true proportions are usually unknown, so agreement with an assumed or inferred reference cannot by itself establish accuracy. Synthetic mixtures provide a controlled test: construct spots from source profiles in known proportions, run the deconvolution procedure, and compare its estimates with those known proportions.

The test answers a specific question: when the data are generated under a stated mixture design, how well does this procedure recover the known fractions? It does not establish performance for every tissue or technical condition. The value of the exercise depends on whether the synthetic data resemble the sources of variation that matter in the intended application.

Constructing a Synthetic Mixture

A useful design begins by stating what the target proportion means. If \(p_{ik}\) is a cell fraction, the mixture generator must represent cell contributions in a way that corresponds to cell fractions. Mixing normalized expression profiles in those proportions assumes, for example, that each cell type contributes the same expected captured RNA per cell. If cell types have different RNA content or capture efficiency, a simple weighted average of normalized profiles may instead represent RNA contributions, not cell fractions. The mixture design should account for that difference or define its target accordingly.

One illustrative count-generation model uses a normalized expression profile \(q_{kg}\) for cell type \(k\) and gene \(g\), with \(\sum_g q_{kg}=1\). Under equal expected captured RNA contribution per cell, the expected gene share in spot \(i\) is \(\pi_{ig}=\sum_k p_{ik}q_{kg}\). Conditional on a chosen library size \(N_i\), counts can be generated as a multinomial draw. This is a controlled mechanism, not a claim that all spatial counts follow a multinomial model; overdispersed or cell-level generation may be preferable when those features are part of the intended test.

For a realistic challenge, vary the known proportions across spots, include pure or near-pure spots if relevant, and vary depth and cell-type combinations. Generate multiple independent replicate data sets. Keep the source profiles used to construct mixtures separate from any data used to tune the deconvolution method; otherwise, the evaluation can reward memorization of the test mixtures. If the intended use includes batch effects, reference mismatch, or spatial structure, include controlled versions of those features and report their results separately rather than blending them into one score.

Summarizing Error Without Hiding Its Direction

For each cell type, calculate performance across the \(n\) synthetic spots. The mean bias measures signed error, mean absolute error (MAE) measures average error magnitude, and root mean squared error (RMSE) penalizes large errors more heavily. Report these separately by cell type before giving any overall summary: a single pooled value can hide a type that is consistently difficult to recover.

All three measures are expressed in fraction units. Multiply by 100 to express them in percentage points. For example, a bias of \(0.04\) means estimates exceed truth by 4 percentage points on average. Bias near zero is not evidence of small errors: positive and negative errors can cancel.

Why Bias and RMSE Tell Different Stories

The distinction follows from an exact decomposition. Let \(e_i=\hat p_{ik}-p_{ik}\) for one cell type and let \(\bar e=(1/n)\sum_i e_i\). Write each error as \(e_i=(e_i-\bar e)+\bar e\). Squaring and averaging gives the empirical mean squared error:

The last equality holds because \(\sum_i(e_i-\bar e)=0\). Thus RMSE squared equals the empirical variance of the errors, using divisor \(n\), plus squared bias. A method can have near-zero bias but substantial variability, or consistent estimates with a small spread but appreciable bias. Reporting more than one metric makes those patterns visible.

Calibration and Boundary-Specific Checks

A method is calibrated over a range when spots assigned similar estimated fractions have, on average, similar true fractions. For example, estimates near 0.30 should correspond to an average true fraction near 0.30. Assess this by plotting true fractions against estimates and, when there are enough spots, grouping estimates into prespecified or clearly reported bins. In a calibration plot, the ideal reference is the identity line. Calibration is not the same as precision: averaging within a bin can conceal large spot-level errors.

Boundary cases deserve their own summaries because overall averages are often dominated by interior fractions. Define the boundary groups before examining results—for example, near-zero truth as \(p_{ik}\leq0.05\) and near-one truth as \(p_{ik}\geq0.95\). Within each group, report the number of spots and relevant errors, such as mean bias, MAE, and the distribution of estimates. For near-zero truth, also report how often the method returns an estimate above a prespecified practical threshold. For near-one truth, inspect how often it substantially underestimates. Thresholds should reflect the application and be fixed before comparing methods.

Diagnostics, Comparisons, and Limitations

Useful diagnostics include plots of estimates against truth with the identity line, error against truth, and error against library size. Stratify or facet results by cell type and by generation condition, such as depth or reference source. A trend in residuals across truth values may indicate shrinkage toward the center or compression near zero and one. A depth trend suggests performance depends on sampling. Large errors concentrated in particular cell-type combinations suggest that the challenge is not captured by an overall score.

Synthetic mixtures made by combining profiles from observed single cells retain some empirical expression patterns, but can still be optimistic if the same cells or highly similar profiles are used for fitting and testing. Parametric simulations make it easier to vary assumptions and isolate a failure mechanism, but results depend on the chosen model. Physical mixtures can include laboratory effects absent from digital mixtures, yet their true cell fractions and mixing process may be less exact. These approaches answer related but distinct questions; report how the mixtures were produced rather than treating them as interchangeable evidence.

A synthetic evaluation is limited by its generator. If the generator omits reference mismatch, background RNA, unequal capture, or technical variation, good recovery under that design does not show robustness to those conditions. Conversely, a deliberately difficult simulation is informative only if the difficulty is relevant and its source is documented. Evaluate multiple designs, show per-type results, and avoid selecting only favorable metrics. When estimates are constrained to valid proportions, errors across types are linked; a high estimate for one type can force lower estimates for others. Interpret per-type results in that context rather than treating them as independent outcomes.