Partial Pooling Across Donors

Why Model Donors Together?

In single-cell RNA sequencing, a donor-specific estimate might summarize a cell-type contrast in expression, a change in cell abundance, or another quantity calculated from that donor’s cells. The estimate varies partly because donors differ biologically and partly because it is measured with finite precision. A hierarchical model represents both sources of variation. It uses information across donors without requiring their underlying effects to be identical.

The goal is not to make every donor look alike. It is to estimate donor-specific effects while acknowledging that they are drawn from a population with its own mean and variability. This differs from treating every cell as an independent biological replicate: the unit in the model below is the donor-level estimate. The distinction complements the earlier material on random-intercept models.

Definition: Let $y_i$ be the estimated effect for donor $i$, with known or estimated sampling variance $v_i$. A Gaussian two-level model is $$ \theta_i \sim N(\mu,\tau^2), \qquad y_i \mid \theta_i \sim N(\theta_i,v_i), \qquad i=1,\ldots,m. $$ Here $\theta_i$ is the donor’s underlying effect, $\mu$ is the population mean effect, and $\tau^2$ is the between-donor variance. The variance $v_i$ describes uncertainty in measuring donor $i$'s effect. The model assumes donor effects are independent draws from the stated population distribution, and that the donor-level estimates are conditionally independent with the specified means and variances.

The model separates two questions: how much donors’ underlying effects vary, represented by $\tau^2$, and how uncertain each measured donor estimate is, represented by $v_i$. For example, $v_i$ may differ across donors because they contribute different numbers of cells or have different within-donor variability. More cells can improve an estimate’s precision, but they do not create additional independent donors.

Three Pooling Strategies

The three approaches differ in how they relate the donor-specific effects. No pooling estimates each $\theta_i$ separately from $y_i$, without using information about other donors. In the simple model, its estimate is $y_i$. Complete pooling assumes all donors share one effect, so $\theta_i=\mu$ for every donor; with independent estimates of known variances, the common estimate is the inverse-variance weighted mean. Partial pooling allows donor effects to vary around $\mu$, while using the population distribution to inform each estimate.

Approach	Donor effects	Information shared across donors
No pooling	Separate, unrelated effects	None for donor-specific estimates
Complete pooling	One common effect	All estimates contribute to the same effect
Partial pooling	Different effects drawn from a population	Donor estimates inform the population mean and variation

Complete pooling can be too restrictive when genuine donor heterogeneity is present. No pooling can be unstable when some donor estimates are noisy. Partial pooling provides a middle ground, provided the distributional assumptions are reasonable and the data contain enough information to estimate the population structure.

Deriving the Donor-Specific Estimate

First suppose $\mu$ and $\tau^2$ are known. Under the Gaussian model, the conditional distribution of $\theta_i$ given $y_i$ is normal. Its mean is the partial-pooling estimate. The result follows by combining the normal population distribution with the normal sampling distribution:

p(\theta_i \mid y_i) \propto \exp\left\{-\frac{(\theta_i-\mu)^2}{2\tau^2} -\frac{(y_i-\theta_i)^2}{2v_i}\right\}.

Collect the terms involving $\theta_i$. The coefficient of $\theta_i^2$ in the exponent is determined by $1/\tau^2+1/v_i$, so the conditional variance is the reciprocal of this sum. Completing the square gives:

\theta_i \mid y_i \sim N\left( \frac{v_i\mu+\tau^2 y_i}{\tau^2+v_i}, \frac{\tau^2v_i}{\tau^2+v_i} \right).

Result: Conditional on $\mu$ and $\tau^2$, the posterior mean of the donor effect is $$ \tilde{\theta}_i =\frac{v_i}{\tau^2+v_i}\mu+\frac{\tau^2}{\tau^2+v_i}y_i =\mu+\frac{\tau^2}{\tau^2+v_i}(y_i-\mu). $$ It is a precision-dependent combination of the population mean and the donor estimate. This expression is also the best linear predictor under the stated Gaussian model.

The formula clarifies the role of the model’s components. If $v_i$ is small relative to $\tau^2$, then $y_i$ is precise compared with the amount of real donor variation, and $\tilde{\theta}_i$ is close to $y_i$. If $v_i$ is large relative to $\tau^2$, then a noisy estimate contributes less information about the donor’s underlying effect. If $\tau^2=0$, the model says all true donor effects are identical, and the estimate reduces to $\mu$. In practice, $\mu$ and $\tau^2$ are usually estimated from the donors; substituting estimates gives an empirical Bayes estimate or a related mixed-model estimate. That substitution adds uncertainty not included in the conditional variance above.

Worked Example: Comparing All Three Approaches

Suppose three hypothetical donors have estimated effects $y_1=1$, $y_2=2$, and $y_3=5$ log-expression units, each with sampling variance $v_i=1$ (standard error 1 log-expression unit). For illustration, take a fitted population mean $\mu=8/3$ and between-donor variance $\tau^2=1.5$ in squared log-expression units.

No pooling: the donor estimates remain $1$, $2$, and $5$ log-expression units.

Complete pooling: with equal variances, the common estimate is the arithmetic mean: $$ \bar{y}=\frac{1+2+5}{3}=\frac{8}{3}=2.667 $$ log-expression units. This approach assigns that same estimate to each donor.

Partial pooling: the data weight is $$ \frac{\tau^2}{\tau^2+v_i}=\frac{1.5}{1.5+1}=0.6, $$ and the population-mean weight is $1-0.6=0.4$. Thus $$ \tilde{\theta}_i=0.4(2.667)+0.6y_i. $$ For the three donors, this gives $1.667$, $2.267$, and $4.067$ log-expression units, respectively. The estimates remain donor-specific but are less extreme than the unpooled estimates. These are conditional estimates treating the stated population parameters as fixed; they are not intervals or tests of donor differences.

Unequal Precision Across Donors

Partial pooling is particularly useful when donor estimates have different sampling variances. The same amount of observed deviation from the population mean should not necessarily have the same influence when one estimate is precise and another is noisy. The conditional formula accounts for this by assigning each donor’s data the weight $\tau^2/(\tau^2+v_i)$.

Worked Example: Estimates With Different Standard Errors

Consider three hypothetical donor effects in log-expression units, with population mean $\mu=0.5$ and between-donor variance $\tau^2=1$ squared log-expression units. Donor A has $y_A=1.5$ and standard error $0.5$, donor B has $y_B=2$ and standard error $1$, and donor C has $y_C=-1$ and standard error $0.5$. Their sampling variances are $0.25$, $1$, and $0.25$, respectively.

For donor A, the data weight is $1/(1+0.25)=0.8$, so $$ \tilde{\theta}_A=0.5+0.8(1.5-0.5)=1.3 $$ log-expression units.

For donor B, the data weight is $1/(1+1)=0.5$, so $$ \tilde{\theta}_B=0.5+0.5(2-0.5)=1.25 $$ log-expression units.

For donor C, the data weight is again $0.8$, so $$ \tilde{\theta}_C=0.5+0.8(-1-0.5)=-0.7 $$ log-expression units. Donors A and C have more precise estimates, so their estimates retain a larger fraction of their observed deviations from $\mu$. This comparison relies on the stated variances and population parameters; it does not establish that any observed donor difference is statistically significant.

Estimating the Population Variation

The population parameters are rarely known. Their estimation is a central part of fitting the hierarchical model, especially when the number of donors is small. A simple method-of-moments calculation illustrates the distinction between observed variability and true between-donor variability. If donors have equal sampling variance $v$, then under the model the marginal variance of $y_i$ is $\tau^2+v$. Therefore, the sample variance of donor estimates tends to include both sources of variation.

\operatorname{Var}(y_i)=\operatorname{Var}(\theta_i)+\operatorname{Var}(y_i\mid\theta_i) =\tau^2+v.

A basic method-of-moments estimate is consequently $s_y^2-v$, truncated at zero because a variance cannot be negative. This is a simple illustration, not a universal fitting rule: likelihood-based or Bayesian methods may be used, and estimation can be difficult with few donors.

Worked Example: A Simple Between-Donor Variance Estimate

Suppose four hypothetical donors have estimates $1$, $2$, $3$, and $4$ log-expression units, each with standard error $0.5$, so $v=0.25$ squared log-expression units. Their mean is $$ \bar{y}=\frac{1+2+3+4}{4}=2.5 $$ log-expression units. The sample variance is $$ s_y^2=\frac{(1-2.5)^2+(2-2.5)^2+(3-2.5)^2+(4-2.5)^2}{4-1} =\frac{5}{3}=1.667 $$ squared log-expression units. The method-of-moments estimate is $$ \hat{\tau}^2=\max(0,1.667-0.25)=1.417 $$ squared log-expression units.

Using $\bar{y}=2.5$ and this estimate, the data weight is $1.417/(1.417+0.25)=0.85$. The resulting conditional empirical Bayes estimates are $2.5+0.85(y_i-2.5)$, or $1.225$, $2.075$, $2.925$, and $3.775$ log-expression units. This calculation treats the estimated population parameters as fixed when producing these donor estimates. With only four donors, $\hat{\tau}^2$ is itself highly uncertain; the arithmetic does not make the fitted population variance precise.

Assumptions, Diagnostics, and Limitations

A partial-pooling estimate is only as credible as the model that produced it. Check the following before interpreting donor-specific estimates:

Donors are the independent units. Dependence between donors, such as relatedness or repeated sampling, may require a different dependence structure.
Sampling variances are meaningful. The values $v_i$ should reflect uncertainty in the donor summaries. Treating highly uncertain estimates as precise distorts the balance between donor data and the population distribution.
The population distribution is plausible. A normal distribution for donor effects is a modeling assumption, not a consequence of having several donors. Strong skewness, outliers, or distinct subpopulations can make a single normal distribution misleading.
The mean structure is appropriate. If donors differ systematically by a relevant group or covariate, a single common $\mu$ may confound that structure with unexplained donor variation.
There is enough donor-level information. When there are few donors, estimates of $\tau^2$ may be unstable or land at zero. The many cells within a donor do not replace missing donor replication for estimating population variation.

Useful diagnostics include comparing observed donor estimates with the fitted population distribution, checking whether residual deviations are much larger or more structured than expected, and examining whether fitted $\tau^2$ is near zero or is implausibly large. Plotting donor estimates with their uncertainty alongside the fitted population mean can reveal unequal precision or extreme observations. Such checks are warnings, not automatic tests that certify a model as correct.

A common mistake is to say, “The donors have different observed estimates, so no pooling is safest.” The correction is that observed differences combine true donor variation and sampling error. No pooling ignores the shared population structure, while complete pooling assumes away all true differences. Partial pooling models both sources, but its estimates can still be misleading if the population distribution, variance estimates, or donor independence assumptions are poor. It also does not by itself solve confounding, selection bias, or limited generalizability beyond the sampled donors.

In practice, the choice among approaches should reflect the scientific target. If the target is one shared effect and donor effects are plausibly identical, complete pooling can be appropriate. If the target is a distinct estimate for every observed donor and no common population model is defensible, no pooling may be preferable. If donors are viewed as varying members of a population and their estimates have unequal precision, partial pooling expresses that structure directly. None of these approaches turns cell-level measurements into independent biological replicates.

Key takeaway: Partial pooling models donor effects as varying around a population mean. It combines each donor’s estimate with that mean according to the donor’s measurement uncertainty and the estimated between-donor variance. The result lies between the assumptions of complete pooling and no pooling, but depends on a defensible hierarchical model.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $v_i$ is very small relative to $\tau^2$, which estimate receives more weight: $y_i$ or $\mu$?
For $\mu=0$, $\tau^2=4$, $y_i=3$, and $v_i=1$, calculate the partial-pooling estimate.
What does complete pooling assume about the donor-specific effects?
Why can many cells from a small number of donors still leave $\tau^2$ poorly estimated?
If a method-of-moments calculation gives $s_y^2-v<0$, what does truncating the estimate at zero mean, and what should be reported as a limitation?

Solutions. (1) The donor estimate receives more weight: $\tau^2/(\tau^2+v_i)$ is near 1 when $v_i$ is small relative to $\tau^2$. (2) The data weight is $4/(4+1)=0.8$, so $\tilde{\theta}_i=0+0.8(3-0)=2.4$ in the same units as the effects. (3) It assumes all donors share one underlying effect, with no between-donor variance. (4) The between-donor variance is learned from variation across donors; additional cells mainly improve the precision of estimates within the observed donors. (5) Truncation sets the estimated variance to zero because variance cannot be negative. It signals that the observed spread does not exceed the modeled sampling variation in this calculation, but with few donors the estimate may be unstable and does not prove that true donor heterogeneity is absent.

What You'll Learn