Spatial Priors for Deconvolution

Why Borrow Information Across Spots?

Deconvolution estimates the mixture of cell types contributing to each spatial spot. A spot with limited counts or weak markers may have a noisy estimate, even when nearby spots have similar tissue composition. A spatial prior formalizes the idea that neighboring spots may have related cell-type proportions, allowing them to inform one another. The benefit is reduced local noise; the risk is that real tissue boundaries and abrupt transitions may be blurred.

Let $i=1,\ldots,n$ index spots and $k=1,\ldots,K$ index cell types. Write $p_{ik}$ for the proportion of cell type $k$ at spot $i$, and let $\hat p_{ik}$ denote an estimate obtained from the spot’s expression and reference information before spatial regularization. At each spot, proportions are nonnegative and sum to one. A neighborhood graph records which pairs of spots are considered spatial neighbors.

Definition: A spatial prior for deconvolution is a probability model that assigns higher prior density to proportion maps in which connected spots have similar cell-type proportions. A common quadratic form is proportional to $$ \exp\left\{-\frac{\lambda}{2}\sum_{(i,j)\in E}w_{ij}\sum_{k=1}^{K}(p_{ik}-p_{jk})^2\right\}, $$ where $E$ is the set of graph edges, $w_{ij}\geq 0$ is the weight of the edge between spots $i$ and $j$, and $\lambda\geq 0$ controls the strength of spatial regularization. The prior is used together with a spot-level likelihood or objective, and the estimates must remain valid proportions.

The penalty is small when strongly connected spots have similar proportions and large when they differ. An edge with weight zero contributes nothing. Increasing $\lambda$ makes spatial similarity more influential relative to the spot-level data. This prior does not assert that all neighboring spots truly have identical composition; it expresses a preference that must be balanced against the observed data.

From a Prior to a Regularized Estimate

A useful working approximation is that the spot-level estimates are approximately Gaussian around the unknown proportions, with known variances $s_{ik}^{2}$. In this approximation, the negative log-likelihood, up to a constant, is $$ \frac{1}{2}\sum_{i=1}^{n}\sum_{k=1}^{K}\frac{(p_{ik}-\hat p_{ik})^2}{s_{ik}^{2}}. $$ Combining this data-fit term with the spatial prior gives a maximum a posteriori (MAP) estimate by minimizing

\frac{1}{2}\sum_{i=1}^{n}\sum_{k=1}^{K}\frac{(p_{ik}-\hat p_{ik})^2}{s_{ik}^{2}} +\frac{\lambda}{2}\sum_{(i,j)\in E}w_{ij}\sum_{k=1}^{K}(p_{ik}-p_{jk})^2

subject to $p_{ik}\geq 0$ and $\sum_{k=1}^{K}p_{ik}=1$ for every spot $i$. A more exact count-based likelihood can replace the Gaussian approximation; the spatial penalty still represents the same borrowing principle. The approximation is most defensible when spot-level estimates are reasonably concentrated and approximately symmetric. It may be poor for highly uncertain estimates near zero or one.

If the likelihood is Gaussian, the prior density above is understood on the set of valid proportion maps. Because this set is compact and the difference penalty is finite and bounded for finite weights and $\lambda$, its exponential has a finite, positive integral and can be normalized to a proper prior. The difference penalty alone does not distinguish maps that are constant across the graph, so it may provide no information about their overall levels.

A Two-Spot Derivation

The shrinkage can be seen directly with one cell type’s proportion at two neighboring spots. Suppose the independent estimates are $y_1$ and $y_2$, with Gaussian error variances $1/a$ and $1/b$, respectively. Let $\lambda$ be the penalty strength for their single edge. Ignoring the other cell type for this scalar illustration, minimize

L(p_1,p_2)=\frac{a}{2}(p_1-y_1)^2+\frac{b}{2}(p_2-y_2)^2+\frac{\lambda}{2}(p_1-p_2)^2.

Differentiating with respect to each estimate and setting the derivatives to zero yields

(a+\lambda)p_1-\lambda p_2=ay_1,\qquad -\lambda p_1+(b+\lambda)p_2=by_2.

These equations show the mechanism: each fitted value is pulled toward its neighbor, while the data term pulls it toward its own estimate. If $a,b>0$, the determinant of the coefficient matrix is $(a+\lambda)(b+\lambda)-\lambda^2=ab+\lambda(a+b)>0$, so the unconstrained quadratic has a unique minimizer. A fitted value still has to satisfy the proportion constraints in the full problem.

Worked Example: Solving Two-Spot Shrinkage Equations

Suppose two neighboring spots have independent estimates $y_1=0.7$ and $y_2=0.1$ for a particular cell type. Let the data precisions be $a=4$ and $b=1$, and set $\lambda=1$. These hypothetical estimates and precisions give the objective

L(p_1,p_2)=2(p_1-0.7)^2+\frac{1}{2}(p_2-0.1)^2+\frac{1}{2}(p_1-p_2)^2.

Differentiating gives the equations $5p_1-p_2=2.8$ and $2p_2-p_1=0.1$. The second equation implies $p_2=(0.1+p_1)/2$. Substituting into the first gives

5p_1-\frac{0.1+p_1}{2}=2.8 \quad\Longrightarrow\quad 4.5p_1=2.8+\frac{0.1}{2}=2.85

This arithmetic must retain the minus sign from subtracting $p_2$: equivalently, multiplying the first substituted equation by two gives $10p_1-0.1-p_1=5.6$, so $9p_1=5.7$. Thus $p_1=0.6333$, and $p_2=(0.1+0.6333)/2=0.3667$. Both are between zero and one. Relative to the independent estimates, the higher estimate moves down and the lower one moves up, toward each other. In a full two-cell-type model, the complementary proportions at these spots must also be fitted so each spot sums to one.

Neighborhoods, Weights, and Boundaries

The graph is part of the model, not a neutral bookkeeping choice. On a regular array, an edge may connect spots sharing a side; other designs use a specified set of nearby spots. The choice should reflect the physical sampling layout and the scale at which borrowing is plausible. If an edge joins spots across an anatomical boundary, a strong penalty can transfer composition from one tissue region into another.

Weights can reduce this problem. For example, an edge weight can be made smaller when spots are separated by a large physical distance or have features suggesting a boundary. Such weights need a defensible basis: if weights are chosen from the same noisy signal being regularized, the apparent boundary may itself be unstable. A uniform graph is simpler and often a useful starting point, but it assumes that every included edge is equally suitable for smoothing.

Worked Example: The Strength of a Boundary Edge

Consider two hypothetical neighboring spots whose estimated proportions of a cell type differ by $0.60$. Under the quadratic prior, the edge contributes $\lambda w_{ij}(0.60)^2/2$ to the penalty. For $\lambda=1$ and a uniform weight $w_{ij}=1$, the contribution is

\frac{1}{2}(1)(0.60)^2=0.18.

If independent evidence supports treating this edge as a likely tissue boundary and its weight is reduced to $0.05$, its contribution becomes

\frac{1}{2}(1)(0.05)(0.60)^2=0.009.

The second specification penalizes the same difference twenty times less. This calculation compares the prior penalty, not the final fitted proportions: the likelihood and all other edges also affect the estimate. It illustrates why a boundary-aware graph can preserve a transition better than uniform smoothing, while also showing that an overly small weight could prevent useful borrowing if the spots are in fact similar.

Worked Example: Smoothing a Noisy Three-Spot Chain

Suppose three spots form a line, with hypothetical independent estimates $0.80,\ 0.75,\ 0.15$ for one cell type. Use unit data precision, equal edge weights of one, and $\lambda=0.5$. The first two spots are similar, while the third is lower. For this scalar Gaussian approximation, minimizing the data-fit plus edge penalty gives the equations

1.5p_1-0.5p_2=0.80,\qquad -0.5p_1+2p_2-0.5p_3=0.75,\qquad -0.5p_2+1.5p_3=0.15.

Multiplying through by two gives $3p_1-p_2=1.60$, $-p_1+4p_2-p_3=1.50$, and $-p_2+3p_3=0.30$. From the first and third equations, $p_2=3p_1-1.60$ and $p_3=(p_2+0.30)/3$. Substitution into the middle equation gives $p_1=0.7467$; then $p_2=0.6400$ and $p_3=0.3133$. The fitted values remain in the valid range for this scalar illustration. Compared with $0.80,\ 0.75,\ 0.15$, the first two move downward, with the second moving farther, so their difference increases; the third moves upward substantially. If the low value at spot 3 is a true boundary rather than noise, that upward movement is undesirable. The example shows why a smaller $\lambda$, a boundary-aware edge, or no edge across that transition may be preferable.

Assumptions, Diagnostics, and Alternatives

Spatial regularization is most reasonable when nearby spots tend to have similar proportions at the chosen spatial scale, the graph encodes meaningful neighbors, and the spot-level likelihood captures the information in the expression data adequately. It does not correct a mismatched reference, uninformative markers, or a misspecified count model. Those issues can make the data-fit estimates systematically wrong, and smoothing can spread that error across the graph.

Useful diagnostics include mapping each cell type’s independently fitted and regularized proportions; inspecting whether sharp transitions coincide with plausible tissue structure; and checking the residual agreement between the fitted proportions and the expression data. Refit over a range of $\lambda$ values and graph definitions. If conclusions change sharply with small tuning changes, the spatial pattern is not robustly determined. Also inspect isolated spots, irregular boundaries, and regions with sparse or uncertain data. A smooth-looking map is not, by itself, evidence of accurate deconvolution.

Independent fitting ($\lambda=0$) avoids spatial bias but may leave substantial noise. A uniform quadratic penalty is computationally convenient and has a transparent shrinkage interpretation, but it can blur abrupt transitions. A boundary-adaptive penalty changes edge weights to allow larger differences where justified, at the cost of estimating or specifying those weights. A prior on transformed proportions, such as log-ratios, can respect positivity through its parameterization, but requires care with zero proportions and still requires a choice of spatial structure. None of these approaches removes the need to examine the data likelihood and neighborhood assumptions.

The regularized MAP estimate is a point estimate, not a complete account of uncertainty. The uncertainty tutorial’s central lesson remains relevant: spots with weak information can have wide uncertainty even when a spatial penalty gives them a stable-looking fitted value. A strong prior may narrow posterior uncertainty, but it does so by imposing structure, not by adding independent measurements. Spatial priors are therefore best treated as controlled information sharing whose assumptions are checked, not as a guarantee that neighboring proportions are equal.

Common Mistake

A common mistake is to interpret a smooth proportion map as proof that the underlying tissue composition varies smoothly. The mistaken reasoning is: “The regularized estimates change gradually, so the tissue must change gradually.” But gradual change is partly built into the penalty. The correction is to compare with independent estimates, examine the likelihood fit and tissue boundaries, and assess sensitivity to $\lambda$ and edge weights. If a transition disappears only under strong smoothing, that is evidence of sensitivity to the prior, not evidence against the boundary.

Key takeaway: A spatial prior trades some local data fit for similarity among connected spots. Its value depends on whether the graph and smoothing scale reflect the tissue; tune and diagnose it with the possibility of real boundaries in mind.

Check Your Understanding

Answer each question, then compare with the solutions below.

What does increasing $\lambda$ do in a quadratic spatial prior, and what is the main risk?
Two neighboring spots have a proportion difference of $0.4$, with $\lambda=2$ and edge weight $0.5$. What is the edge penalty contribution?
Why is a smooth fitted map not sufficient evidence that the true tissue composition is smooth?
Name two diagnostics for deciding whether spatial regularization is appropriate.

Solutions. (1) Increasing $\lambda$ strengthens the preference for neighboring spots to have similar proportions; the main risk is erasing genuine boundaries. (2) The contribution is $\lambda w_{ij}(0.4)^2/2=2(0.5)(0.16)/2=0.08$. (3) Smoothness is encouraged by the prior itself, so the appearance may reflect regularization rather than the underlying tissue. (4) Suitable checks include comparing independent and regularized maps, inspecting fit to expression data and boundaries, and refitting over several $\lambda$ values or graph definitions.