Deriving Posterior Mean Shrinkage

From a Test Statistic to an Effect Estimate

A test statistic summarizes evidence relative to a null model, but it is not itself an estimate of the underlying effect on the original measurement scale. To estimate an effect, we can model the observed estimate as a noisy measurement of a latent effect and combine that likelihood with a prior that allows some effects to be exactly zero. The resulting posterior mean has two sources of shrinkage: uncertainty about whether the effect is nonzero, and uncertainty about its size conditional on being nonzero.

Suppose $Y$ is an observed effect estimate, such as an estimated log fold change, and $s$ is its known standard error. The latent effect is $\theta$. The model below uses a point mass at zero, called the spike, and a normal distribution for nonzero effects, called the slab. It is a specific model for effect estimates; it is not a formula that applies to every test statistic or prior.

Definition: Let $\gamma$ indicate whether the effect is nonzero. The spike-and-slab normal model is $$ \begin{aligned} &P(\gamma=0)=\pi_0,\qquad P(\gamma=1)=1-\pi_0,\\ &\theta=0\quad\text{if }\gamma=0,\qquad \theta\mid\gamma=1\sim N(0,\tau^2),\\ &Y\mid\theta\sim N(\theta,s^2). \end{aligned} $$ Here $0<\pi_0<1$, $\tau^2>0$, and $s^2>0$. The sampling variance $s^2$ is treated as known, and the prior and sampling model are assumed correctly specified. Conditional on the slab, the prior is centered at zero; the model therefore shrinks positive and negative observations symmetrically.

The posterior probability $P(\gamma=0\mid Y=y)$ is the local false discovery rate for this observation in the two-group model. Write $q(y)=P(\gamma=1\mid Y=y)$ for the posterior probability that the effect is nonzero. In the formulas below, $\pi_0$ and $\tau^2$ are model inputs; in an empirical Bayes analysis they may have been estimated from many tests.

Deriving the Posterior Mean

First condition on being in the slab. The likelihood as a function of $\theta$ is proportional to $\exp\{-(y-\theta)^2/(2s^2)\}$, and the slab prior is proportional to $\exp\{-\theta^2/(2\tau^2)\}$. Multiplying these densities gives a normal posterior after completing the square:

\begin{aligned} -\frac{(y-\theta)^2}{2s^2}-\frac{\theta^2}{2\tau^2} &=-\frac{1}{2}\left(\frac{1}{s^2}+\frac{1}{\tau^2}\right) \left(\theta-\frac{\tau^2}{s^2+\tau^2}y\right)^2+C(y),\\ \theta\mid Y=y,\gamma=1 &\sim N\left(\frac{\tau^2}{s^2+\tau^2}y,\, \frac{s^2\tau^2}{s^2+\tau^2}\right). \end{aligned}

The constant $C(y)$ does not depend on $\theta$, so it affects neither the posterior mean nor the posterior variance. The conditional posterior mean is $k y$, where $k=\tau^2/(s^2+\tau^2)$. This factor is between zero and one: a small slab variance relative to sampling variance produces stronger shrinkage toward zero.

The posterior probability of the slab is obtained from Bayes' rule. Under the spike, $Y\sim N(0,s^2)$. Under the slab, integrating over $\theta\sim N(0,\tau^2)$ gives $Y\sim N(0,s^2+\tau^2)$. Denote the corresponding normal densities by $f_0(y)$ and $f_1(y)$. Then

q(y)=P(\gamma=1\mid Y=y) =\frac{(1-\pi_0)f_1(y)} {\pi_0 f_0(y)+(1-\pi_0)f_1(y)}.

This posterior probability weights the slab-conditional estimate by the chance that the slab generated the observation. Under the spike, the effect is exactly zero. Applying the law of total expectation therefore yields the posterior mean:

Result: Under the stated model, $$ E(\theta\mid Y=y) =q(y)\frac{\tau^2}{s^2+\tau^2}y. $$ The estimate combines the posterior probability of a nonzero effect, $q(y)$, with the within-slab shrinkage factor, $\tau^2/(s^2+\tau^2)$. The posterior mean has the same sign as $y$, or is zero when $y=0$.

For calculation, the posterior odds of a nonzero effect can be written as prior odds times a Bayes factor:

\frac{q(y)}{1-q(y)} =\frac{1-\pi_0}{\pi_0}\frac{f_1(y)}{f_0(y)},\qquad \frac{f_1(y)}{f_0(y)} =\sqrt{\frac{s^2}{s^2+\tau^2}} \exp\left\{\frac{y^2}{2} \left(\frac{1}{s^2}-\frac{1}{s^2+\tau^2}\right)\right\}.

The Bayes factor depends on $y^2$, so for this symmetric model the posterior probability of a nonzero effect is the same for $y$ and $-y$. The posterior mean, in contrast, changes sign. These formulas also show why the raw effect estimate alone is not enough: the standard error, prior nonzero proportion and slab scale all affect the result.

Worked Example: Separating the Two Shrinkage Factors

Suppose a hypothetical gene has observed log fold-change estimate $y=2.4$ and standard error $s=0.8$. Use $\pi_0=0.90$ and slab standard deviation $\tau=1.2$. Assume the standard error is known and the spike-and-slab normal model is appropriate for this illustration.

The null and alternative marginal variances are $s^2=0.64$ and $s^2+\tau^2=0.64+1.44=2.08$. The Bayes factor for the slab is

\frac{f_1(2.4)}{f_0(2.4)} =\sqrt{\frac{0.64}{2.08}} \exp\left\{\frac{2.4^2}{2} \left(\frac{1}{0.64}-\frac{1}{2.08}\right)\right\} \approx 12.50.

Prior odds for a nonzero effect are $0.10/0.90=0.1111$. Thus posterior odds are $0.1111(12.50)=1.389$, giving $q(2.4)=1.389/(1+1.389)\approx0.5814$. The conditional slab shrinkage factor is $1.44/(0.64+1.44)=0.6923$. Therefore

E(\theta\mid Y=2.4) =0.5814(0.6923)(2.4)\approx0.966.

The posterior mean is about $0.966$ log fold-change units. It is smaller than the observed estimate both because the posterior probability of a nonzero effect is below one and because the slab-conditional estimate is itself shrunk toward zero.

Worked Examples: Prior Evidence and Measurement Precision

Worked Example: A Small Observation With a Concentrated Slab

For a hypothetical effect estimate, let $y=0.5$, $s=1$, $\pi_0=0.80$, and $\tau=0.5$. The observation model has standard error 1, and the slab is relatively concentrated around zero. Both are compatible with the assumed model for this calculation.

Here $s^2=1$ and $s^2+\tau^2=1.25$. The Bayes factor is

\frac{f_1(0.5)}{f_0(0.5)} =\sqrt{\frac{1}{1.25}} \exp\left\{\frac{0.5^2}{2}\left(1-\frac{1}{1.25}\right)\right\} \approx 0.9171.

The prior odds of a nonzero effect are $0.20/0.80=0.25$. Posterior odds are $0.25(0.9171)=0.2293$, so $q(0.5)=0.2293/(1+0.2293)\approx0.1865$. Within the slab, the shrinkage factor is $0.25/1.25=0.20$. The posterior mean is therefore

E(\theta\mid Y=0.5)=0.1865(0.20)(0.5)\approx0.0187.

The estimate is close to zero: the observation gives little support for the slab relative to the spike, and even conditional on the slab, the prior concentrates effects near zero. This small posterior mean is not evidence that the true effect is known to be zero; it is the mean under the specified model and inputs.

Worked Example: The Same Estimate With Different Precision

Compare two hypothetical measurements, each with $y=2$, under the same prior $\pi_0=0.90$, $\tau=1$. The only difference is the standard error. Assume the normal sampling model and the same prior are suitable for both.

For $s=1$, the marginal variances are 1 and 2. The Bayes factor is

\frac{f_1(2)}{f_0(2)} =\sqrt{\frac{1}{2}}\exp\left\{\frac{2^2}{2} \left(1-\frac{1}{2}\right)\right\} \approx1.922.

The prior odds are $1/9$, so posterior odds are $1.922/9=0.2136$, and $q(2)\approx0.1760$. The conditional shrinkage factor is $1/(1+1)=0.5$. Hence the posterior mean is $0.1760(0.5)(2)\approx0.176$.

For $s=0.5$, the marginal variances are $0.25$ and $1.25$. The Bayes factor is

\frac{f_1(2)}{f_0(2)} =\sqrt{\frac{0.25}{1.25}} \exp\left\{\frac{2^2}{2} \left(\frac{1}{0.25}-\frac{1}{1.25}\right)\right\} \approx269.2.

Posterior odds are $(1/9)(269.2)\approx29.91$, giving $q(2)\approx0.9677$. The conditional shrinkage factor is $1/1.25=0.8$, so the posterior mean is $0.9677(0.8)(2)\approx1.548$. Thus identical raw estimates can yield quite different posterior means when their measurement precision differs.

Assumptions, Diagnostics, and Alternatives

The derivation relies on a normal sampling distribution with the specified standard error, a point mass exactly at zero, and a zero-centered normal slab with variance $\tau^2$. For independent tests, applying the model test by test also requires that the effect estimates and their standard errors are interpreted on compatible scales. In practice, the prior parameters may be estimated rather than known; the calculations above condition on fixed values and do not include uncertainty in those estimates.

Diagnostics should examine whether the model's implied marginal distribution is plausible, especially near the center and in the tails, and whether the assumed null center and scale fit the data. Unusually heavy tails, systematic shifts, or different behavior across groups can signal that the normal slab or null model is inadequate. Sensitivity checks that vary $\pi_0$ and $\tau$ are also useful: if posterior means change substantially, the estimates depend strongly on prior specification. A small posterior mean alone cannot diagnose which assumption is failing.

This posterior mean differs from two common alternatives. The mean conditional on $\gamma=1$ is $k y$, which ignores the posterior probability that the effect is exactly zero. A hard-threshold rule instead reports zero or a nonzero estimate according to a cutoff; it discards the graded uncertainty represented by $q(y)$. Neither alternative is generally the posterior mean under this model. The posterior mean is useful as a squared-error summary, but it can be close to zero even when the posterior distribution has meaningful probability away from zero. Reporting the estimate alongside uncertainty or posterior probabilities may therefore be important.

Common mistake: Multiplying the observed estimate by only $\tau^2/(s^2+\tau^2)$ and calling the result the spike-and-slab posterior mean. That is the mean conditional on the slab. The overall posterior mean also multiplies by $q(y)$, because the spike assigns probability to an exactly zero effect.

Check Your Understanding

Use the spike-and-slab normal model above. Solutions are included so you can check both the calculation and the interpretation.

For $y=1$, $s=1$, $\pi_0=0.5$, and $\tau=1$, calculate the Bayes factor, $q(y)$, and the posterior mean.
Under the stated model, how does the posterior mean change if $y$ is replaced by $-y$, with all other quantities fixed?
Explain why $k y$ is not, by itself, the overall posterior mean in a spike-and-slab model.
For fixed $s$ and $\tau$, what happens to the within-slab shrinkage factor as $\tau^2$ decreases? Give the interpretation.

Solutions. (1) The Bayes factor is $\sqrt{1/2}\exp\{(1/2)(1-1/2)\}\approx0.9079$. Prior odds are 1, so $q(1)=0.9079/(1+0.9079)\approx0.4759$. Since $k=1/(1+1)=0.5$, the posterior mean is $0.4759(0.5)(1)\approx0.2380$. (2) The posterior slab probability is unchanged because the Bayes factor depends on $y^2$, while the conditional mean changes sign. The posterior mean therefore changes sign as well. (3) $k y$ conditions on the effect being nonzero; the overall mean weights that estimate by $q(y)$, with the remaining posterior probability assigned to the point mass at zero. (4) The factor $k=\tau^2/(s^2+\tau^2)$ decreases as $\tau^2$ decreases. A more concentrated slab pulls the slab-conditional estimate more strongly toward zero.

Deriving Posterior Mean Shrinkage

What You'll Learn