Likelihoods for RNA and Antibody-Derived Tag Counts

Why the Count Likelihood Depends on the Modality

A CITE-seq experiment measures RNA and antibody-derived tags in the same cells, but the two count matrices do not record the same biological quantity. An RNA count for a gene summarizes captured and sequenced molecules assigned to that gene. An antibody-derived tag (ADT) count summarizes captured and sequenced tags associated with an antibody, used as a measurement related to a surface protein. Both are nonnegative integers, but that common format alone does not make their distributions, scales, or interpretations interchangeable.

A likelihood makes the assumed count-generating distribution explicit. It describes how plausible the observed counts are under specified model parameters. Poisson and negative-binomial (NB) likelihoods are useful starting points for count data, provided their assumptions are made clear. In particular, the NB model allows more variance than the Poisson model at the same mean.

A likelihood is not a probability distribution over parameter values: its values do not generally sum or integrate to one over parameters. Nor does a large likelihood by itself say that a model fits well in an absolute sense. Likelihoods are useful for estimating parameters and comparing specified models, but those tasks require attention to assumptions and fit.

Poisson Likelihoods and Exposure

The Poisson model is a natural baseline when a count is generated by events occurring independently at a fixed rate over a known exposure. For sequencing counts, it is usually a working model for the observed counts, rather than a complete physical description of every stage of library preparation and sequencing. Its central restriction is that the conditional variance equals the conditional mean.

The mean and variance are both \(\mu\). If counts are conditionally independent across cells, the likelihood is the product of their Poisson probabilities. For RNA, a common basic specification is \(Y_{ig}\sim\operatorname{Poisson}(\mu_{ig})\). One may express the mean as \(\mu_{ig}=s_i\lambda_{ig}\), where \(s_i\) is a known exposure or size factor for cell \(i\), and \(\lambda_{ig}\) is the expected count per unit of that exposure. This form assumes that the chosen offset adequately represents relative exposure; it does not make \(s_i\) a measurement of absolute RNA content.

For ADTs, a corresponding baseline is \(A_{ih}\sim\operatorname{Poisson}(\nu_{ih})\), with \(\nu_{ih}\) the expected count for tag \(h\) in cell \(i\). This is a distributional statement about tag counts, not a claim that ADT counts measure protein abundance on the same scale as RNA counts measure gene expression. Parameters and offsets must be defined for the modality and analysis at hand.

Deriving a Poisson Rate Estimate

Suppose \(C_i\) are independent Poisson counts with means \(s_i\lambda\), where the exposures \(s_i>0\) are known and \(\lambda>0\) is the shared rate per exposure unit. Independence and the Poisson distribution are assumptions of this derivation. The likelihood and log-likelihood are

Terms involving \(\lambda\) can be differentiated to find the maximum. Setting the derivative to zero gives

The second derivative is \(-\sum_i C_i/\lambda^2\), which is negative when the total count is positive, so this stationary point is a maximum. If all counts are zero, the likelihood is maximized at the boundary \(\hat{\lambda}=0\). This derivation shows how exposure enters the rate estimate: counts are pooled relative to total exposure, not averaged as if every cell had the same exposure.

Negative-Binomial Likelihoods for Extra-Poisson Variation

Single-cell counts often vary more across cells than a Poisson model permits, even after accounting for modeled covariates and exposure. The NB distribution provides a mean-variance relationship that allows such extra-Poisson variation. One standard parameterization uses mean \(\mu>0\) and dispersion \(\alpha>0\). In this convention, larger \(\alpha\) means greater variance at a fixed mean.

Here \(\Gamma(\cdot)\) is the gamma function, which satisfies \(\Gamma(k)=(k-1)!\) for positive integers \(k\). Under this parameterization, \(E(C)=\mu\) and \(\operatorname{Var}(C)=\mu+\alpha\mu^2\). The Poisson variance is recovered in the limit as \(\alpha\) approaches zero, with \(\mu\) fixed. Thus, the NB model relaxes the Poisson equality of mean and variance, but it still imposes a specific distributional shape.

For RNA, one may specify \(Y_{ig}\) as NB with mean \(\mu_{ig}\) and gene-specific dispersion \(\alpha_g\); for ADTs, one may specify \(A_{ih}\) as NB with mean \(\nu_{ih}\) and tag-specific dispersion \(\phi_h\). These are separate model choices. A common formulation links the means to covariates and exposure, for example \(\log(\mu_{ig})=\log(s_i)+x_i^\mathsf{T}\beta_g\), where \(x_i\) is a vector of observed predictors and \(\beta_g\) is a coefficient vector. This assumes the log-mean relationship is appropriate, the offset is valid, and the conditional count distribution is NB. It does not guarantee those assumptions.

If counts are assumed conditionally independent, the joint likelihood across cells and features is a product of the corresponding mass functions. In practice, independence is a modeling assumption conditional on included quantities; shared cell-level effects, unmodeled covariates, or dependence among measurements can make the product likelihood an imperfect description.

Comparing Likelihoods Across RNA and ADT Counts

The same count value can have different interpretations and model parameters across modalities. Consider a hypothetical cell with an RNA count of 4 for one gene and an ADT count of 4 for one tag. A Poisson model with mean 3 for the RNA count assigns

For comparison, an NB model for the ADT count with mean 3 and dispersion \(\alpha=0.2\) has size \(r=5\). Its probability at 4 is

The observed count is 4 in both cases, but the probabilities differ because the likelihoods and their parameters differ. Even if the numerical probabilities had matched, that would not make the RNA and ADT measurements equivalent. Their count-generating processes, feature definitions, and scientific meanings differ.

Assumptions, Diagnostics, and Limitations

A Poisson likelihood assumes that the conditional mean is correctly specified, the variance equals that mean, and the relevant counts are conditionally independent under the model. For an exposure-offset model, it also assumes the offset captures the relevant proportional exposure. An NB likelihood relaxes the variance-equals-mean restriction, but still assumes its mean structure and NB count distribution are adequate. Both models can be affected by omitted predictors, dependence, measurement effects, or systematic differences among cells.

Useful diagnostics compare observed and model-implied features, such as the mean and variance across comparable cells, the frequency of zeros, and residual patterns against fitted means or covariates. A variance much larger than the fitted Poisson mean is evidence against a simple Poisson specification, though it does not identify the cause. Persistent residual structure, or a mismatch between predicted and observed count frequencies, can indicate a poor mean model or a distributional mismatch. Dispersion estimated near zero in an NB model suggests that the data provide little evidence for extra-Poisson variance under that fitted model; it does not prove the Poisson assumptions.

The Poisson model is simpler and can be useful when its conditional variance assumption is plausible. The NB model is a common alternative when extra variation remains, at the cost of estimating dispersion parameters. Neither model is a universal solution: an NB distribution does not absorb every kind of heterogeneity or dependence, and an offset cannot repair a misspecified mean structure. These likelihoods also do not, by themselves, resolve questions about the biological meaning of a count, comparability of cells, or the adequacy of a measurement process.

Applying a Count Likelihood Carefully

A practical specification can be organized as a sequence of explicit decisions. This helps distinguish a chosen probability model from claims about the underlying biology.