Regularized Dispersion for Stable Gene Rankings

Why Regularize Dispersion Estimates?

A gene’s observed variability across cells combines biological differences with sampling noise. When only a small number of cells contribute information for a gene, its estimated dispersion can fluctuate considerably: one or two unusually large counts may make it appear highly variable. Ranking genes directly by these noisy estimates can therefore elevate genes whose apparent variability is largely uncertain.

Regularization addresses this instability by pulling a gene-wise dispersion estimate toward a mean-dispersion trend: the expected dispersion for genes with similar mean expression. The pull is stronger when the gene’s estimate is less precise. This is a compromise, not a claim that every gene follows the trend exactly. The earlier material on mean-variance trends and binned dispersion motivates the trend and the problem; here the focus is how shrinkage changes estimates and rankings.

Definition: Under a negative-binomial model with mean $\mu_j$ and dispersion $\alpha_j$, gene $j$'s counts have variance $\mu_j+\alpha_j\mu_j^2$. A mean-dispersion trend $\alpha_0(\mu_j)$ gives the typical dispersion at that mean. A regularized dispersion estimate is a weighted average of the gene-wise estimate and the trend value: $$ \tilde{\alpha}_j=w_j\hat{\alpha}_j+(1-w_j)\alpha_0(\mu_j),\qquad 0\leq w_j\leq 1. $$ Here $\hat{\alpha}_j$ is the gene-wise estimate and $w_j$ is its reliability weight. A smaller $w_j$ means stronger shrinkage toward the trend.

The model is a working description of count variability, not a requirement that all cells have identical expected counts. In a simple equal-exposure illustration, $Y_{ij}$ denotes the count for gene $j$ in cell $i$, with mean $\mu_j$. In an analysis that accounts for cell-specific depth, the mean may vary across cells through offsets or fitted means; dispersion estimation should then respect that fitted model. The simple calculations below assume equal exposure, a common mean within each gene, comparable cells, and a trend value treated as already estimated. They are hypothetical illustrations, not a recommendation to fit a trend from four genes or four cells.

From Gene-Wise Dispersion to a Regularized Score

For equal-exposure counts, a moment estimate follows by matching the negative-binomial variance to the sample variance. If $\bar{Y}_j$ is the sample mean and $s_j^2$ the sample variance for gene $j$, the untruncated moment estimate is $(s_j^2-\bar{Y}_j)/\bar{Y}_j^2$. Since dispersion cannot be negative, a simple estimate is

\hat{\alpha}_j=\max\left(0,\frac{s_j^2-\bar{Y}_j}{\bar{Y}_j^2}\right).

This formula is useful for understanding the calculation. In practice, a fitted negative-binomial model may account for unequal expected counts and other features of the design. The formula also makes clear why sparse, small samples can be troublesome: sample variance is itself noisy, and estimates at or near zero are common.

A regularized ranking can use $\tilde{\alpha}_j$, or compare it with the trend by using the ratio $\tilde{\alpha}_j/\alpha_0(\bar{Y}_j)$, when the trend is positive. A ratio near 1 indicates dispersion close to the trend; a ratio above 1 indicates higher estimated dispersion. The trend-adjusted comparison matters because raw dispersion generally depends on mean expression. A large raw dispersion does not by itself mean a gene is unusually variable for its expression level.

Why Shrinkage Can Stabilize an Estimate

The basic justification is a bias-variance trade-off. Suppose, for a moment, that a gene-specific quantity $\hat{\theta}$ is unbiased for a true value $\theta$, has variance $\tau^2$, and is shrunk toward a fixed target $\theta_0$ using $\tilde{\theta}=w\hat{\theta}+(1-w)\theta_0$. Then

\operatorname{Bias}(\tilde{\theta})=(1-w)(\theta_0-\theta),\qquad \operatorname{Var}(\tilde{\theta})=w^2\tau^2.

The mean squared error is variance plus squared bias:

\operatorname{MSE}(\tilde{\theta}) =w^2\tau^2+(1-w)^2(\theta-\theta_0)^2.

Differentiating with respect to $w$, setting the derivative to zero, and solving gives $w=(\theta-\theta_0)^2/[\tau^2+(\theta-\theta_0)^2]$. Thus, when estimation noise $\tau^2$ is large relative to the gene’s departure from the target, a smaller weight can reduce expected squared error. In real analyses, neither the true departure nor the variance is known, so weights are estimated or specified using a model. The calculation explains the rationale; it does not identify one universally correct weight.

For a transparent teaching example, one can use $w_j=n_{\mathrm{det},j}/(n_{\mathrm{det},j}+\kappa)$, where $n_{\mathrm{det},j}$ is the number of cells with a detected count and $\kappa>0$ controls the amount of shrinkage. This is a simple heuristic, not a standard law of dispersion estimation: detection count is only a rough proxy for information. Real methods may use model-based uncertainty, likelihoods, or empirical-Bayes estimates. The governing principle is to shrink less when the gene-wise estimate is more reliable.

Worked Example: Shrinking One Gene-Wise Estimate

Suppose four hypothetical, comparable cells have counts $0,0,2,6$ for a gene. Assume equal exposure and a fitted trend value of $\alpha_0(2)=0.4$ at the gene’s mean. Use the moment estimator above and the illustrative weight $w=n_{\mathrm{det}}/(n_{\mathrm{det}}+\kappa)$ with $\kappa=2$.

The sample mean is $\bar{Y}=(0+0+2+6)/4=2$. The squared deviations from 2 sum to $4+4+0+16=24$, so the sample variance is $s^2=24/(4-1)=8$. The unregularized estimate is

\hat{\alpha}=\max\left(0,\frac{8-2}{2^2}\right)=\frac{6}{4}=1.5.

Two cells have nonzero counts, so $w=2/(2+2)=0.5$. The regularized estimate is

\tilde{\alpha}=0.5(1.5)+0.5(0.4)=0.75+0.20=0.95.

Relative to the trend, the raw estimate is $1.5/0.4=3.75$, whereas the regularized estimate is $0.95/0.4=2.375$. It remains above trend, but the amount of estimated excess dispersion is moderated. This calculation assumes the supplied trend is appropriate and the simple equal-exposure model is adequate; four observations alone cannot establish either assumption.

Worked Example: A Small-Cell Ranking Reverses

Consider two hypothetical genes measured in the same four equal-exposure cells. Suppose the fitted trend values, supplied from a larger set of genes, are $0.5$ for gene A at mean 2 and $0.05$ for gene B at mean 1. Use $\kappa=1$ and rank by dispersion relative to trend.

For gene A, the counts are $0,0,0,8$. Their mean is 2, and the squared deviations sum to $4+4+4+36=48$, giving $s^2=48/3=16$. Therefore,

\hat{\alpha}_A=\frac{16-2}{2^2}=3.5,\qquad \frac{\hat{\alpha}_A}{\alpha_0(2)}=\frac{3.5}{0.5}=7.

Only one cell has a nonzero count, so $w_A=1/(1+1)=0.5$, and

\tilde{\alpha}_A=0.5(3.5)+0.5(0.5)=2.0,\qquad \frac{\tilde{\alpha}_A}{\alpha_0(2)}=\frac{2.0}{0.5}=4.

For gene B, the counts are $0,0,2,2$. The mean is 1, and $s^2=(1+1+1+1)/3=4/3$. Thus $\hat{\alpha}_B=(4/3-1)/1^2=1/3$, and the raw ratio is $(1/3)/0.05=6.667$. Two cells are detected, giving $w_B=2/(2+1)=2/3$. The regularized estimate and ratio are

\tilde{\alpha}_B=\frac{2}{3}\left(\frac{1}{3}\right)+\frac{1}{3}(0.05) =\frac{2}{9}+\frac{1}{60}=0.2389,\qquad \frac{\tilde{\alpha}_B}{0.05}=4.778.

The unregularized ratios rank A above B (7 versus 6.667), while the regularized ratios rank B above A (4.778 versus 4). The reversal arises because A’s more extreme estimate is based on a single detected cell and receives stronger shrinkage. It does not prove B is biologically more variable: the result depends on the trend, the weight rule, and the observations.

Worked Example: A Zero Moment Estimate Is Not Evidence of No Variability

Suppose a third hypothetical gene has counts $0,1,1,2$ in four comparable cells, with mean 1 and trend value $0.2$. The squared deviations from 1 sum to 2, so $s^2=2/3$. The raw moment estimate is truncated at zero:

\hat{\alpha}=\max\left(0,\frac{2/3-1}{1^2}\right)=0.

With three detected cells and $\kappa=3$, the illustrative weight is $w=3/(3+3)=0.5$. Therefore $\tilde{\alpha}=0.5(0)+0.5(0.2)=0.1$, and the regularized ratio is $0.1/0.2=0.5$. The estimate is below trend, not proof that the true dispersion is zero. Sampling variability and truncation explain why a zero estimate can occur.

Assumptions, Diagnostics, and Alternatives

Regularization is most defensible when genes being compared have estimates on a comparable scale, the mean-dispersion trend captures the broad relationship in the data, and the reliability weights reflect actual information. It also assumes that pulling estimates toward that trend is useful for the ranking goal. If a biologically meaningful subgroup of genes departs systematically from the trend, strong shrinkage can suppress that signal.

Check the analysis rather than treating a regularized ranking as automatically stable. Inspect whether the trend follows the broad pattern across expression levels, whether a small number of genes or cells dominate its fit, and whether estimates change substantially when reasonable trend or weight choices are varied. Also check that low-expression genes were handled appropriately and that remaining differences are not simply due to depth or technical structure. A trend fitted to a narrow or distorted gene set may pull estimates toward the wrong target.

Approach	Strength	Limitation
Unregularized gene-wise dispersion	Preserves each gene’s estimate without shrinkage.	Can produce unstable rankings when estimates are noisy.
Binned dispersion	Compares genes within groups of similar mean expression.	Results can depend on bin boundaries and may be unstable within a bin.
Regularized dispersion	Moderates noisy gene-wise estimates using a mean-dispersion trend.	Can over-shrink genuine departures or inherit a poorly fitted trend.

These approaches answer related but not identical ranking questions. Binning groups genes before comparison; regularization adjusts a gene-wise estimate toward a trend. Neither approach removes the need to consider the expression scale, filtering, and data quality. Regularized dispersion is a ranking device, not a hypothesis test: a high rank is not a p-value, a confidence statement, or evidence of a causal biological effect.

Common Mistakes

Wrong: “The largest raw dispersion must rank first.” Correction: dispersion depends on mean expression; compare estimates to an appropriate mean-dispersion trend.
Wrong: “Shrinkage makes every estimate more accurate.” Correction: shrinkage can lower noise but adds bias when a gene truly departs from the target.
Wrong: “A zero moment estimate means the gene has no biological variation.” Correction: the estimate can be zero because the sample variance is below the Poisson component or because of truncation.
Wrong: “The detection-based weight is universally correct.” Correction: it is a teaching heuristic; weights should reflect the precision assumptions of the method being used.

Key takeaway: Regularized dispersion combines a noisy gene-wise estimate with the expected dispersion at that gene’s mean. The amount of shrinkage should reflect reliability, and rankings should be interpreted alongside the trend, weight choices, and biological context.

Check Your Understanding

Try each question, then compare your reasoning with the solutions.

If $\hat{\alpha}_j=0.8$, $\alpha_0(\mu_j)=0.3$, and $w_j=0.4$, calculate $\tilde{\alpha}_j$.
Why might an unregularized estimate based on one detected cell be less reliable than one based on several detected cells?
In the ranking example, why did gene A move below gene B after regularization?
What does a regularized dispersion ratio of 1 indicate, and what does it not establish?

Solutions. 1. $\tilde{\alpha}_j=0.4(0.8)+0.6(0.3)=0.32+0.18=0.50$. 2. One detected cell provides little information about variability across cells, so its estimate may be strongly affected by a single count; detection count is only a rough proxy for precision. 3. A’s larger raw ratio was based on one detected cell, so its smaller reliability weight pulled it more strongly toward trend; B’s ratio was moderated less. 4. A ratio of 1 means the regularized estimate equals the trend value at that mean. It does not show that the gene is unimportant, that its true dispersion exactly equals the trend, or that a statistical hypothesis has been tested.