Why Reference Mismatch Matters
Spatial deconvolution estimates which cell types contribute to each measured spot by comparing spot expression with cell-type profiles in a reference. The interpretation depends on a substantive assumption: the reference must represent the cell types and expression states contributing to the spots. If a contributing type is absent, or a represented type has a different expression state in the tissue, the fitted proportions can absorb expression differences that are not actually compositional. The result may be a plausible-looking proportion map that is systematically biased.
The aim here is to diagnose that problem, not to treat every large residual as proof of a particular missing cell type. A mismatch can leave a detectable residual, but it can also lie in the span of the reference profiles and be absorbed into the fitted proportions. Residual inspection and sensitivity analysis therefore provide evidence about model adequacy, not automatic identification of the true tissue composition.
Model and Sources of Mismatch
For a spot, let \(y\) be a vector of expression measurements for \(G\) genes, and let \(B\) be a \(G \times K\) reference matrix whose \(k\)th column is the profile for reference cell type \(k\). Let \(\theta\) be the vector of its proportions, with \(\theta_k \geq 0\) and \(\sum_{k=1}^K\theta_k=1\). In a linear mixture formulation, the spot is modeled as a weighted combination of reference profiles plus error.
Here \(d\) is the systematic discrepancy between the true mean spot profile and the mean profile represented by \(B\theta\), and \(\varepsilon\) is sampling or measurement noise with mean zero conditional on the reference and the true composition. This formulation is for expression on a scale where a linear mixture is appropriate, with any required library-size or exposure scaling handled consistently. For count data, the corresponding count model and its assumptions must be used; a linear residual analysis should not be applied to raw counts as though they had constant variance.
The linear mixture requires that the reference and spot measurements use compatible genes, normalization, and scale; that the profiles represent the relevant cell types and states; and that the mixture approximation is reasonable. Estimating distinct proportions also requires the reference profiles to be sufficiently distinguishable on the genes used. If two columns of \(B\) are identical, their separate proportions cannot be identified from this model. Even when profiles differ, near-collinearity can make estimates highly sensitive to noise and reference perturbations.
How Discrepancy Enters the Estimate
For a fixed reference, consider first the unconstrained weighted least-squares estimator. Assume \(W\) is a fixed positive-definite weight matrix and \(B\) has full column rank. The estimator minimizes the weighted squared residual over unrestricted coefficient vectors. Substituting the mixture model into that estimator separates the effect of discrepancy from the effect of zero-mean noise.
Taking expectation over \(\varepsilon\), conditional on \(B,W,d,\theta\), gives the expectation for this unconstrained estimator. The mismatch contributes the component of \(d\) that the fitted reference can represent, expressed in coefficient coordinates.
The remaining component of \(d\), which cannot be represented by changing coefficients, appears in the residual. With simplex constraints, however, the estimator is a constrained projection and is generally a nonlinear function of \(y\). Noise can move a fit onto a boundary, so the displayed expectation formula does not generally hold for the constrained estimator. The displayed formula is exact for the unrestricted estimator. For a simplex-constrained estimator, an interior fit still obeys the sum-to-one constraint, so its expectation generally requires the equality-constrained estimator's formula; the unrestricted formula applies only when the two estimators coincide.
Worked Example: An Omitted Type Leaves a Residual
Suppose two reference profiles for two genes are \(A=(8,2)^\mathsf{T}\) and \(B=(2,8)^\mathsf{T}\). A third type, omitted from the reference, has profile \(C=(8,8)^\mathsf{T}\). In a hypothetical spot, the true mixture is \(0.4A+0.3B+0.3C\). Assume these measurements are on a compatible linear-mixture scale and ignore noise for this calculation.
The observed profile is
The two-type reference model must assign proportions \(t\) to \(A\) and \(1-t\) to \(B\), where \(0\leq t\leq1\). Its fitted profile is \((2+6t,8-6t)^\mathsf{T}\). Under ordinary least squares, the unconstrained minimizing value is
This value lies in the allowed interval, so it is also the constrained fit. The predicted profile is \((5.3,4.7)^\mathsf{T}\), giving residual \(y-\widehat y=(0.9,0.9)^\mathsf{T}\) and unweighted residual sum of squares \(0.9^2+0.9^2=1.62\). The fitted proportions, 0.55 for \(A\) and 0.45 for \(B\), are not the true fractions 0.4 and 0.3; the missing 0.3 contribution has no corresponding reference column. The positive residual in both genes signals a profile component the two-type model cannot reproduce, but does not by itself identify its biological source.
Residuals as Evidence, Not a Verdict
For a fitted spot, the gene-level residual is \(r=y-\widehat y\). Its size and structure are more informative than a single overall fit statistic. A large residual for a gene means that the fitted mixture underpredicts or overpredicts that gene on the analysis scale. Similar residual directions across multiple spots may point to a shared missing expression program or state difference. Residuals that vary systematically across tissue regions can indicate that the reference mismatch itself varies spatially.
A useful diagnostic sequence is to examine residuals by gene and spot, then ask whether their patterns recur across spots with similar anatomy or state. If the fitting procedure uses weights, inspect residuals on a scale that accounts for those weights or the assumed observation variance. Under a well-specified model, systematic structure should not remain in the residuals after accounting for modeled mean and variance. But random noise, normalization artifacts, and other model misspecification can also produce structure; a residual pattern is a reason to investigate, not a unique diagnosis.
The converse is equally important: small residuals do not certify correct proportions. If an omitted profile is representable by a combination of included profiles, the model may fit well while reallocating its contribution among the included types. In geometric terms, an unrepresented expression component can be absorbed if it lies within the convex hull of the reference profiles.
Worked Example: State Mismatch Can Shift Proportions
Suppose the reference profiles are \(A=(8,2)^\mathsf{T}\) and \(B=(2,8)^\mathsf{T}\). The actual spot contains equal amounts of the two types, but type \(A\) is in a tissue state with profile \((10,2)^\mathsf{T}\) rather than its reference profile. Assume no noise and fit the reference with unweighted least squares and simplex constraints.
The observed profile is \(y=0.5(10,2)^\mathsf{T}+0.5(2,8)^\mathsf{T}=(6,5)^\mathsf{T}\). The reference fit at proportion \(t\) for \(A\) is \((2+6t,8-6t)^\mathsf{T}\). Projecting \(y-(2,8)^\mathsf{T}=(4,-3)^\mathsf{T}\) onto the direction \((6,-6)^\mathsf{T}\) gives
The fitted profile is \((2+6(0.5833),8-6(0.5833))^\mathsf{T}\approx(5.5,4.5)^\mathsf{T}\); the residual is approximately \((0.5,0.5)^\mathsf{T}\). The estimate assigns about 58.3% to \(A\), although the hypothetical composition was 50% of each type. The discrepancy has both a component that shifts the fitted proportion and a component left in the residual. A state-aware reference with \(A=(10,2)^\mathsf{T}\) would reproduce \(y\) at \(t=0.5\). This comparison illustrates why proportions should be interpreted conditional on the reference, not as direct measurements independent of it.
Sensitivity Analysis and Alternatives
A sensitivity analysis asks whether the scientific conclusion changes under plausible alternative reference assumptions. It should vary one defensible feature at a time where possible, keep the gene set and fitting scale fixed, and compare both the estimated proportions and residual behavior. Candidate perturbations include adding a plausible missing profile, substituting a tissue-state-specific profile for a generic one, or fitting separately with alternative reference profiles. These are diagnostic comparisons, not a license to select whichever reference produces a preferred biological result.
Record the reference profiles, normalization, gene set, fitting objective, constraints, and uncertainty procedure. These choices define what the estimated proportions mean.
Check gene-level and spot-level residuals, including whether their signs or magnitudes recur across locations or suspected tissue states.
For example, compare the baseline reference with a justified added type or a profile representing a relevant tissue state. Do not change unrelated choices at the same time if the goal is to isolate sensitivity.
Summarize changes in proportions alongside residual changes and uncertainty. A better fit with substantially different estimates indicates reference dependence, not proof that the alternative is correct.
Worked Example: Testing an Omitted-Type Explanation
Return to the hypothetical data \(y=(6.2,5.6)^\mathsf{T}\) and the reference profiles \(A=(8,2)^\mathsf{T}\), \(B=(2,8)^\mathsf{T}\). The two-type fit from the first example had estimates 0.55 and 0.45 and residual sum of squares 1.62. Now compare a plausible augmented reference that also includes \(C=(8,8)^\mathsf{T}\). The data were constructed as \(0.4A+0.3B+0.3C\), so this augmented model can fit exactly with proportions \((0.4,0.3,0.3)\).
Indeed, \(0.4A=(3.2,0.8)^\mathsf{T}\), \(0.3B=(0.6,2.4)^\mathsf{T}\), and \(0.3C=(2.4,2.4)^\mathsf{T}\); their sum is \((6.2,5.6)^\mathsf{T}\). The augmented fit therefore has residual \((0,0)^\mathsf{T}\) and residual sum of squares 0. In this constructed example the additional profile both improves fit and restores the specified mixture. In real data, an improved fit alone would not establish that a proposed type is present: extra flexibility can reduce residuals, and the augmented profile must be justified independently.
The main alternatives are to retain the original reference while reporting its limitations, use a biologically justified reference that better represents the tissue, or introduce an explicit discrepancy component when the analysis supports one. These approaches answer different questions. A reference expansion attributes expression to additional types; a state-specific reference changes the profile for a represented type; a discrepancy term allows unmodeled expression without necessarily assigning it to a cell type. Each can affect identifiability. In particular, allowing many flexible discrepancy terms can make cell-type proportions weakly determined.
Failure Modes and Interpretation
- Reading residuals as cell identities. A residual pattern may be consistent with a missing type, but can also reflect state mismatch, normalization error, or an unsuitable variance model. It is evidence to investigate, not a cell-type label.
- Treating a low residual as validation. A mismatch that can be represented by existing profiles may be hidden in biased proportions. Assess robustness to plausible references as well as fit.
- Using an unconstrained bias formula for a constrained fit. The linear expectation expression applies to the unconstrained estimator under its stated assumptions. The simplex-constrained estimator can hit boundaries, so zero-mean noise does not guarantee an unbiased constrained estimate.
- Ignoring weak identifiability. Similar profiles can yield unstable proportion estimates even when residuals are small. Large changes under small reference perturbations are an important warning.
- Confusing RNA contribution with cell abundance. A proportion estimated from expression mixtures may describe relative RNA contributions under the model, not necessarily the fraction of cells, especially when types differ in RNA content or capture efficiency.
Reference mismatch matters because biological conclusions often depend on comparing estimated compositions across spots or regions. If a state shift is mistaken for a change in cell abundance, apparent spatial enrichment may be an artifact of the reference. Report the reference, the scale and interpretation of estimated proportions, residual diagnostics, and the results of plausible sensitivity analyses. When estimates change materially across reasonable references, describe that dependence rather than presenting one map as uniquely determined.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the linear model, what part of a reference discrepancy affects the unconstrained coefficient estimate, and what part remains in the residual?
- Why does zero-mean measurement noise not generally imply an unbiased simplex-constrained estimate?
- A reference change greatly alters estimated proportions but barely changes residual size. What does this suggest?
- Why is a small residual not sufficient evidence that the estimated proportions are correct?
Solutions. (1) The component represented by the columns of \(B\), in the weighted least-squares geometry, shifts the unconstrained coefficients by \((B^\mathsf{T}WB)^{-1}B^\mathsf{T}Wd\); the component not represented by the fitted profiles remains in the residual. (2) The constrained fit is nonlinear in the data: noise can move estimates onto a boundary, so averaging the constrained estimates need not equal the unconstrained linear expectation. (3) The data are compatible with multiple reference-dependent allocations, indicating weak identification or sensitivity to the reference; low residuals do not resolve which allocation is biologically correct. (4) A mismatch can lie within the set of profiles the reference can combine, allowing it to be absorbed into altered proportions without leaving a large residual.