Shrinkage Of Dispersion Estimates

Why Shrink Gene-Wise Dispersion Estimates?

In an RNA-seq negative binomial model, dispersion describes how much the counts vary beyond the variation expected under a Poisson model. For a gene with mean normalized count \(\mu\) and dispersion \(\alpha\), the variance is

A gene-wise estimate of \(\alpha\) is informed by the replicate samples for that gene. With few replicates, however, the data may provide little information about its dispersion. A gene can then receive an estimate that is unusually high or low partly because of sampling noise, rather than because its true variability differs greatly from that of genes with similar expression.

Earlier material on common, trended, and gene-wise dispersion distinguishes a single shared value, a mean-dependent trend, and separate estimates for individual genes. Shrinkage combines the latter two ideas: it keeps gene-specific information, but moderates it toward the shared trend. This is a way of borrowing information across genes, not a claim that all genes have identical dispersion.

What Is Being Combined?

A useful conceptual description works on the log-dispersion scale. Define \(\eta_i=\log(\alpha_i)\), so positive dispersions correspond to real-valued log dispersions. Let \(\hat{\eta}_i=\log(\hat{\alpha}_i)\) and \(\eta_{0i}=\log\{\alpha_0(\mu_i)\}\). A generic shrinkage estimate has the form

Here \(w_i\) represents the relative influence of the gene-wise estimate in this conceptual combination. If the gene has substantial information about its dispersion, \(w_i\) is closer to 1 and the result stays closer to \(\hat{\eta}_i\). If its estimate is noisy, \(w_i\) is smaller and the trend has more influence. The corresponding dispersion is \(\tilde{\alpha}_i=\exp(\tilde{\eta}_i)\). Because the combination is on the log scale, it is a weighted geometric combination on the original dispersion scale, not generally a weighted arithmetic average.

This expression explains the idea, not a universal software formula. Implementations can use likelihood-based procedures, estimate how much dispersion varies around the trend, and account for uncertainty in different ways. In particular, the trend is estimated from many genes; it is not known without error. The precise estimator and its derivation are a separate matter.

Why the Borrowing Can Help

A short justification comes from a simple model for log-dispersion estimates. Suppose a gene's log dispersion differs from the trend by \(u_i\), with variance \(\omega^2\), and its observed log estimate has sampling error \(e_i\), with variance \(s_i^2\). In this simplified model, \(\hat{\eta}_i=\eta_{0i}+u_i+e_i\), where \(u_i\) and \(e_i\) have mean zero and are uncorrelated. The error in the weighted estimate \(\eta_{0i}+w_i(\hat{\eta}_i-\eta_{0i})\) is \(w_i e_i-(1-w_i)u_i\). Its expected squared error is therefore

Differentiating with respect to \(w_i\) and setting the derivative to zero gives \(2w_i s_i^2-2(1-w_i)\omega^2=0\). Solving yields \(w_i=\omega^2/(\omega^2+s_i^2)\). Thus, in this simplified setting, greater sampling uncertainty \(s_i^2\) reduces the weight on the gene-wise estimate, while greater real variation \(\omega^2\) among genes increases it. This illustrates the bias-variance trade-off: moving an estimate toward the trend can add bias for a gene that truly differs from the trend, but can reduce the larger sampling variability of an unmoderated estimate.

Empirical-Bayes-style procedures use information from many genes to estimate features of the shared distribution or trend, then use that shared information when estimating each gene's dispersion. The benefit is greatest when individual genes have weak replication but the collection of genes contains enough information to estimate the overall pattern. Shrinkage does not manufacture extra replicate samples for a gene; it stabilizes its estimate using assumptions about how dispersions relate across genes.

Consequences for Count Variability and Inference

The variance formula shows why a change in dispersion matters. For a hypothetical gene with normalized mean \(\mu=20\), dispersion \(0.40\) implies variance \(20+0.40(20^2)=180\) counts squared. Using the shrunken value \(0.1621\) instead gives \(20+0.1621(20^2)=84.84\) counts squared, to the displayed precision. These calculations illustrate the effect of a dispersion value on the model variance; they do not establish which value is correct for a real gene.

In differential-expression inference, the dispersion estimate contributes to uncertainty calculations. A noisy gene-wise estimate can therefore make inference unstable: an overly high estimate may overstate uncertainty, while an overly low one may understate it. Moderation aims to improve the stability and calibration of inference across genes, particularly when each gene has few replicates. It does not guarantee a smaller standard error or a more significant result for every gene. A low estimate can be moderated upward, and a high one downward.

The target of this procedure is dispersion. It does not shrink the observed counts, the fitted mean, or the gene's log fold change. This distinction matters: dispersion shrinkage stabilizes a component of the count model, whereas effect shrinkage concerns a different quantity.

Assumptions, Diagnostics, and Alternatives

Shrinkage is useful when the gene-wise estimates are noisy and the trend is a reasonable description of how typical dispersion varies with mean expression. It relies on the collection of genes providing relevant shared information: after accounting for the mean-dependent trend, the remaining gene-to-gene differences must be sufficiently comparable for borrowing to help. The negative binomial mean-variance model and the design used to estimate gene-wise dispersions must also be suitable for the data. A trend cannot rescue a systematically misspecified count model.

Useful checks focus on whether the assumptions behind borrowing look plausible. Examine gene-wise estimates in relation to the trend, and ask whether the overall pattern is smooth and supported across the range of means. Look for groups of genes or individual genes that depart substantially from the general pattern, and check whether such departures could reflect data quality, unmodelled structure, or genuine biological variability. A trend based on sparse information at one end of the mean range may be uncertain there. An unusually influential gene can also distort a fitted shared pattern.

• Warning sign: many gene-wise estimates show a systematic pattern away from the fitted trend. This suggests that the trend may not represent the data adequately.
• Warning sign: a gene has an extreme estimate that is important to the scientific conclusion. Strong moderation toward the trend could conceal a real departure, so the gene deserves closer investigation.
• Warning sign: the data contain unexpected variability or design structure that the count model does not capture. Borrowing information does not correct such misspecification.

A common alternative is to use the gene-wise dispersion estimates without moderation. That retains each gene's estimate but can be unstable when replication is weak. A common dispersion uses one value for all genes, which can be stable but ignores differences in dispersion and its relationship with mean expression. A trended dispersion captures the mean-dependent pattern but does not, by itself, use the gene-specific estimate in the same way as shrinkage. Shrinkage is a compromise: it allows individual departures while restraining estimates that are poorly supported by their own data.

No borrowing rule is automatically best. If gene-specific departures are large and well supported, excessive shrinkage can pull estimates too close to the trend. If the shared trend is poorly estimated or inappropriate, shrinkage can propagate its errors across many genes. These limitations are reasons to examine the data and model, not reasons to assume that either raw estimates or moderated estimates are always preferable.