Count Likelihoods for Spatial Expression

Why the Count Likelihood Matters

A spatial expression analysis begins with counts: for a given gene, each spot has a nonnegative integer count, and spots can have very different library sizes. A likelihood makes explicit how a model connects those observations to expected expression. It also makes clear which differences in observed counts are attributed to library size, which to modeled expression, and which are left as random variation.

This tutorial specifies Poisson and negative-binomial likelihoods for spot-level counts. The focus is the count distribution and its library-size offset, not a model for spatial dependence. As discussed in the earlier material on separating expression changes from composition changes, a spot’s observed expression can reflect both its cellular composition and expression within cell types. A count likelihood alone does not distinguish those sources.

The exposure \(L_i\) may be a library-size measure or a normalized size factor, depending on the analysis. Its precise definition matters: the model treats it as given, and \(\exp(\eta_{ig})\) is the expected count per unit of that exposure. If the exposures are rescaled by a common constant, the fitted intercept changes to compensate; therefore, the units and normalization of \(L_i\) affect the interpretation of the rate.

The Poisson Likelihood With an Offset

The Poisson model assumes that, conditional on the modeled mean, the count in each spot follows a Poisson distribution. For a single gene, a common specification is

For example, \(\eta_{ig}=\beta_{0g}+x_i^\mathsf{T}\beta_g\), where \(x_i\) contains spot-level predictors. Conditional independence across spots is also assumed in the basic likelihood. The probability mass function and the full likelihood for the gene are

Taking logarithms gives the log-likelihood

The Poisson distribution has conditional mean and variance both equal to \(\mu_{ig}\). Thus the model assumes that, once predictors and exposure are accounted for, the remaining count variation is no greater than the Poisson variance. This is a substantive assumption, not just a convenient formula.

Negative-Binomial Likelihood and Overdispersion

A negative-binomial model allows the conditional variance to exceed the mean. Use the mean-dispersion parameterization in which \(\mu_{ig}>0\) is the mean and \(\alpha_g\geq0\) is the dispersion. For \(\alpha_g>0\), its probability mass function is

Here \(\Gamma(\cdot)\) is the gamma function. As for the Poisson model, set \(\mu_{ig}=L_i\exp(\eta_{ig})\). The negative-binomial variance is

The dispersion parameter is nonnegative: as \(\alpha_g\) approaches zero, the negative-binomial distribution approaches the Poisson distribution. A larger \(\alpha_g\) permits more variability around the same mean. In this parameterization, \(\alpha_g\) is not the variance itself and is not directly comparable across genes without considering their means.

For independent spots, the negative-binomial likelihood is the product of these probabilities, and its log-likelihood is the sum of their logarithms. Regression coefficients and dispersion are generally estimated numerically; unlike the common-rate Poisson example, there is not generally a simple closed-form estimate for both. The model still assumes that the specified mean is appropriate and, in its basic form, that counts are conditionally independent across spots.

Using the Offset to Compare Rates

The offset separates exposure from rate. If two spots have the same rate but one has twice the library-size exposure, its expected count is twice as large. A model without the offset could mistake that technical exposure difference for an expression difference. A coefficient on \(\log(L_i)\) that is freely estimated is not the standard offset model: it allows the count to scale with library size at a rate other than proportionality.

Assumptions, Diagnostics, and Alternatives

Both likelihoods require a count response, a positive exposure for every modeled spot, and a defensible conditional mean model. The offset should be defined consistently and should represent the intended library-size adjustment. If \(L_i=0\), then \(\log(L_i)\) is undefined and the spot cannot be handled by this offset specification as written. Such a spot requires a principled preprocessing or modeling decision; replacing zero with an arbitrary small number changes the model.

The Poisson model is appropriate when conditional variation is plausibly close to the mean and conditional independence is reasonable for the intended analysis. The negative-binomial model relaxes the mean-equals-variance restriction, but it does not automatically account for dependence between nearby spots. Spatially structured residuals can remain even when the marginal variance is well described by a negative-binomial distribution. Modeling spatial dependence is a separate extension.

Useful checks include comparing observed variability with that implied by fitted means, examining Pearson or deviance residuals for systematic patterns, and checking whether residuals vary with fitted count or library size. A Poisson fit with widespread residual overdispersion suggests that its variance assumption may be inadequate. A negative-binomial fit may address that extra variation, but residual patterns across coordinates can still indicate dependence or a missing spatially patterned predictor. Conversely, a flexible negative-binomial model is not automatically preferable if estimated dispersion is negligible and the additional parameter is unsupported.

A common mistake is to say, “The negative binomial allows overdispersion, so it accounts for spatial clustering.” It does not necessarily do so. Its dispersion allows extra marginal count variation under its specified likelihood; it does not encode which spots are neighbors or how their counts covary. Another mistake is to omit the offset and compare raw counts across spots with unequal library sizes. The correction is to specify the exposure and mean model first, then assess variance and residual dependence separately.