Partial Pooling In Hierarchical Regression

Why Pool Information Across Groups?

Many econometric data sets contain natural groups: regions, firms, schools, or time periods. A regression fitted separately within each group can reflect genuine differences, but estimates from groups with little data may be unstable. A single regression fitted to all observations can be more stable, but it forces the groups to have identical coefficients. Partial pooling offers a model-based compromise: group coefficients may differ, while a shared distribution connects them and lets information from the groups contribute to one another’s estimates.

The amount of pooling is not an arbitrary adjustment made after fitting the model. It follows from the likelihood and the assumed distribution of group coefficients. The formulas below treat the error variance and the distribution’s parameters as specified. This isolates the source and interpretation of shrinkage.

A Normal Hierarchical Regression

Definition: For groups $g=1,\ldots,G$, let $y_g$ be the $n_g$-vector of outcomes and $X_g$ the design matrix for that group. A normal hierarchical regression is $$ y_g\mid \beta_g \sim N(X_g\beta_g,\sigma^2 I_{n_g}),\qquad \beta_g\mid\mu,\Sigma \sim N(\mu,\Sigma). $$ Here $\beta_g$ is the vector of group-specific regression coefficients, $\mu$ is their shared prior mean, and $\Sigma$ describes their between-group variation. Conditional on the group coefficients, the outcome errors are independent normal errors with common variance $\sigma^2$; conditional on $\mu,\Sigma$, the group coefficient vectors follow a common normal distribution. In this tutorial, $\sigma^2,\mu,\Sigma$ are treated as specified, with $\Sigma$ positive definite.

The group-level distribution is a model for how coefficients vary across groups. It does not say that every group has coefficient $\mu$; rather, $\mu$ is the center of the distribution. When $\Sigma$ is small, the model expects groups to be similar and tends to pull their estimates toward $\mu$. When $\Sigma$ is large, it permits greater differences and the data from each group have more influence on its estimate.

For a given group, the posterior distribution of $\beta_g$, conditional on the specified hyperparameters, is normal:

\beta_g\mid y_g,\mu,\Sigma,\sigma^2\sim N(m_g,V_g),\qquad V_g=\left(\frac{X_g^\mathsf{T}X_g}{\sigma^2}+\Sigma^{-1}\right)^{-1},\qquad m_g=V_g\left(\frac{X_g^\mathsf{T}y_g}{\sigma^2}+\Sigma^{-1}\mu\right).

The two terms in the posterior precision $V_g^{-1}$ have a useful interpretation. The term $X_g^\mathsf{T}X_g/\sigma^2$ is the group’s data precision, while $\Sigma^{-1}$ is the prior precision supplied by the shared distribution. The posterior mean combines the corresponding precision-weighted information. This is the multivariate form of partial pooling.

Deriving the Group Posterior

The result follows by multiplying the group’s normal likelihood by its normal prior and collecting the terms that depend on $\beta_g$. Up to a constant independent of $\beta_g$, the log posterior is

-\frac{1}{2\sigma^2}(y_g-X_g\beta_g)^\mathsf{T}(y_g-X_g\beta_g) -\frac{1}{2}(\beta_g-\mu)^\mathsf{T}\Sigma^{-1}(\beta_g-\mu).

Expanding the quadratic terms gives a quadratic coefficient $-\tfrac12\beta_g^\mathsf{T}(X_g^\mathsf{T}X_g/\sigma^2+\Sigma^{-1})\beta_g$ and a linear coefficient $\beta_g^\mathsf{T}(X_g^\mathsf{T}y_g/\sigma^2+\Sigma^{-1}\mu)$. Completing the square identifies the precision matrix as $V_g^{-1}=X_g^\mathsf{T}X_g/\sigma^2+\Sigma^{-1}$, and the posterior mean as $V_g(X_g^\mathsf{T}y_g/\sigma^2+\Sigma^{-1}\mu)$. Since both precision matrices are positive definite or positive semidefinite with $\Sigma^{-1}$ positive definite, $V_g$ is positive definite and the resulting normal distribution is well defined.

Partial Pooling for a Group Mean

The simplest case makes the shrinkage especially clear. Suppose observations in group $g$ satisfy $y_{ig}=\theta_g+\epsilon_{ig}$, where $\epsilon_{ig}\sim N(0,\sigma^2)$, and assume $\theta_g\sim N(\mu,\tau^2)$. With $n_g$ observations, the group sample mean $\bar y_g$ has likelihood $\bar y_g\mid\theta_g\sim N(\theta_g,\sigma^2/n_g)$. Combining this likelihood with the prior gives

E(\theta_g\mid \bar y_g)= \frac{n_g\tau^2}{\sigma^2+n_g\tau^2}\bar y_g+ \frac{\sigma^2}{\sigma^2+n_g\tau^2}\mu, \qquad \operatorname{Var}(\theta_g\mid \bar y_g)= \frac{\sigma^2\tau^2}{\sigma^2+n_g\tau^2}.

The first weight, $\lambda_g=n_g\tau^2/(\sigma^2+n_g\tau^2)$, is the weight on the group’s sample mean. The second weight, $1-\lambda_g$, is the weight on the shared center. For fixed variances, a larger group has a larger $\lambda_g$, so its estimate is less strongly shrunk. Greater between-group variance $\tau^2$ also reduces shrinkage; greater observation noise $\sigma^2$ increases it.

Worked Example: Shrinking a Small Group’s Mean

Suppose a hypothetical group has $n_g=4$ outcome measurements with sample mean $\bar y_g=72$ points. Use a normal prior centered at $\mu=60$ points, with between-group variance $\tau^2=64$ points squared, and suppose the known outcome-error variance is $\sigma^2=100$ points squared. The normal, common-variance assumptions are being made for this calculation.

The weight on the group mean is $$ \lambda_g=\frac{4(64)}{100+4(64)}=\frac{256}{356}=0.7191. $$

Thus the posterior mean is $$ m_g=0.7191(72)+0.2809(60)=51.78+16.85=68.63\text{ points}. $$

The group’s estimate is between its sample mean of 72 points and the shared center of 60 points. The conditional posterior variance and standard deviation are $$ V_g=\frac{100(64)}{100+4(64)}=\frac{6400}{356}=17.98\text{ points}^2,\qquad \sqrt{V_g}=4.240\text{ points}. $$

This posterior uncertainty is conditional on the specified center and variance components. The calculation does not account for uncertainty in those quantities.

Complete Pooling, No Pooling, and Partial Pooling

These three approaches encode different assumptions about group differences. For an intercept-only model, complete pooling uses one common mean for every group; under a common-variance normal model, its estimate is the overall sample mean. No pooling estimates each group mean separately as $\bar y_g$, without connecting groups through a shared distribution. Partial pooling instead estimates group-specific means within the hierarchical model: each estimate is informed by its own observations and by the common distribution.

Approach	Group structure	Implication for estimates
Complete pooling	All groups share the same coefficient.	One common estimate; group differences are not represented.
No pooling	Each group has its own coefficient, estimated separately.	Group estimates use only their own data.
Partial pooling	Group coefficients vary around a shared distribution.	Group estimates differ, but are pulled toward the distribution’s center.

The hierarchical model does not guarantee that partial pooling will always give the best predictions or the most appropriate scientific answer. It makes a substantive assumption: group coefficients are related through a common distribution. Its advantage over the two extremes is that it can represent differences while stabilizing estimates for groups with limited information.

Worked Example: Comparing Estimates Across Unequal Group Sizes

Consider hypothetical outcome data from two groups. Group A has $n_A=4$ observations with mean 60 points; Group B has $n_B=16$ observations with mean 70 points. Assume a common known error variance $\sigma^2=100$ points squared. For the partial-pooling calculation, use a specified prior center $\mu=65$ points and between-group variance $\tau^2=100$ points squared.

The overall sample mean used by the complete-pooling model is $$ \bar y_{\text{overall}}=\frac{4(60)+16(70)}{4+16} =\frac{1360}{20}=68\text{ points}. $$ Complete pooling assigns 68 points to both groups. With no pooling, the estimates remain 60 points for A and 70 points for B.

For Group A, the partial-pooling weight is $4(100)/(100+4(100))=0.8000$, giving $$ m_A=0.8000(60)+0.2000(65)=61.00\text{ points}. $$ For Group B, the weight is $16(100)/(100+16(100))=1600/1700=0.9412$, giving $$ m_B=0.9412(70)+0.05882(65)=69.71\text{ points}. $$

Both partial-pooling estimates move toward the specified center of 65 points, but Group B moves less because it has more observations. The complete-pooling value of 68 points is not the center used in this partial-pooling model: the example specifies a prior center of 65 points rather than estimating that center from these two groups.

Pooling Regression Slopes

The same principle applies to slopes and to several coefficients at once. For a single group’s regression, if the likelihood for its slope estimate $b_g$ is $b_g\mid\beta_g\sim N(\beta_g,v_g)$, and the hierarchical prior is $\beta_g\sim N(\mu,\tau^2)$, then the posterior mean is $$ E(\beta_g\mid b_g)=\frac{v_g}{v_g+\tau^2}\mu+ \frac{\tau^2}{v_g+\tau^2}b_g, \qquad \operatorname{Var}(\beta_g\mid b_g)=\frac{v_g\tau^2}{v_g+\tau^2}. $$

Here $v_g$ is the sampling variance of the group’s slope estimate. A slope estimated imprecisely has larger $v_g$, and therefore receives relatively more weight from the shared center. A precise slope receives relatively more weight from its group-specific data.

Worked Example: Shrinking a Group-Specific Slope

Suppose a hypothetical regression in one group estimates an outcome slope of $b_g=0.80$ outcome units per predictor unit, with known sampling variance $v_g=0.0100$ squared outcome units per squared predictor unit. Let the specified shared prior for the group slope have mean $\mu=0.20$ and variance $\tau^2=0.0400$, in the corresponding slope units squared. Assume the normal likelihood and prior are appropriate.

The weight on the group estimate is $$ \frac{\tau^2}{v_g+\tau^2}=\frac{0.0400}{0.0100+0.0400}=0.8000. $$

The posterior mean slope is therefore $$ m_g=0.8000(0.80)+0.2000(0.20)=0.6400+0.0400=0.6800 $$ outcome units per predictor unit. Its posterior variance and standard deviation are $$ V_g=\frac{(0.0100)(0.0400)}{0.0500}=0.008000,\qquad \sqrt{V_g}=0.08944 $$ in slope units. The group-specific estimate is pulled toward 0.20, while retaining substantial influence from its data. This is a model-based association estimate; it does not by itself establish a causal effect.

Interpretation and Common Pitfalls

A group’s posterior mean is an estimate of that group’s coefficient conditional on the observed group data and the specified hierarchical model. Shrinkage is a consequence of combining the group’s likelihood with the shared distribution; it is not evidence that all groups truly have the same coefficient. The posterior variance quantifies uncertainty about the group coefficient conditional on the model inputs. If the center or variance components are uncertain, treating them as fixed can understate total uncertainty.

Mistake: treating partial pooling as complete pooling. A shrunken estimate is not forced to equal the common center. The group coefficient remains distinct unless the model’s assumptions or limiting weights make it coincide with the center.
Mistake: assuming every group is shrunk by the same amount. The weight depends on the group’s information, such as its sample size or slope-estimation variance. Groups with more precise data are generally shrunk less.
Mistake: interpreting shrinkage as a correction that must improve every estimate. Shrinkage follows from the assumed model and may be inappropriate if groups are not plausibly related through the specified distribution.

Key takeaway: Partial pooling combines group-specific evidence with a shared distribution. The group estimate stays responsive to its own data, while the amount it moves toward the shared center depends on the relative information in the group likelihood and the hierarchical prior.

Check Your Understanding

Work through these questions using the models and assumptions stated above.

In the intercept-only model, what happens to the weight on $\bar y_g$ as $n_g$ increases, holding both variances fixed?
A group’s estimate is much closer to the shared center than its sample mean. Give two model-based reasons this could occur.
In the two-group example, why did Group B’s partial-pooling estimate move less toward 65 points than Group A’s?
Explain why complete pooling and no pooling are limiting conceptual alternatives to partial pooling, rather than the same model with different labels.
What additional source of uncertainty is omitted when the hierarchical parameters are treated as specified?