Spike-And-Slab Priors For Bayesian Variable Selection

Why Put a Probability on Inclusion?

In a regression with many candidate predictors, it is often unclear which coefficients should be exactly zero and which should be estimated as nonzero. Selecting one specification and then reporting only its coefficient estimates hides uncertainty about that choice. A spike-and-slab prior makes inclusion itself part of the probability model: each coefficient has a point mass at zero, alongside a continuous distribution for values when it is included.

This construction gives a posterior probability for each inclusion indicator, as well as a posterior distribution for the coefficient. These answer different questions. The inclusion probability summarizes uncertainty about whether the coefficient is in the model; the conditional coefficient distribution describes its size and sign given inclusion.

Model and Prior Specification

Consider the Gaussian regression model \(y=\alpha\mathbf{1}+X\beta+\varepsilon\), where \(y\) is an \(n\)-vector of outcomes, \(X\) contains \(p\) candidate predictors, and \(\varepsilon\sim N(0,\sigma^2 I_n)\). The intercept \(\alpha\) is included in every specification here. For simplicity, first suppose \(\sigma^2\) is known. The same indicator construction applies when variance is unknown; then the model evidence integrates over its prior as well.

A common specification assigns each indicator an independent Bernoulli prior, \(\gamma_j\sim\mathrm{Bernoulli}(\pi_j)\), and uses a zero-centered normal slab. Conditional on the indicators, coefficients with \(\gamma_j=0\) are fixed at zero, while included coefficients receive the slab distribution. For example, \(\beta_j\mid\gamma_j=1\sim N(0,\tau_j^2)\). The slab variance controls the prior scale of included effects; it does not determine whether a coefficient is included.

Equivalently, the prior for \(\beta_j\) is a mixture with probability \(1-\pi_j\) concentrated exactly at zero and probability \(\pi_j\) distributed over the slab. Write \(\delta_0\) for a point mass at zero. Then the marginal prior is \((1-\pi_j)\delta_0+\pi_jN(0,\tau_j^2)\). With \(p\) indicators there are \(2^p\) possible inclusion configurations, often called models or specifications.

Posterior Inclusion Probabilities

For an indicator configuration \(\gamma=(\gamma_1,\ldots,\gamma_p)\), its marginal likelihood is the probability density of the observed data after integrating out the included coefficients and any other parameters given their priors. Denote it by \(m_\gamma(y)\). Bayes’ rule updates the prior probability of each configuration using this evidence.

The posterior inclusion probability for predictor \(j\) sums the probabilities of all configurations that include it. It is not generally obtained by comparing just two specifications: when there are several predictors, uncertainty about the others must also be accounted for.

For a single indicator, the update has a particularly simple form. Define the Bayes factor \(BF_{10}\) as the marginal likelihood under inclusion divided by the marginal likelihood under exclusion. Posterior odds equal prior odds multiplied by this Bayes factor.

A One-Coefficient Bayes Factor

A closed-form Bayes factor illustrates how the data and slab scale interact. Suppose the response has been residualized with respect to the intercept and any other regressors, and consider one candidate predictor \(x\). Assume the resulting model is \(y=x\beta+\varepsilon\), with \(\varepsilon\sim N(0,\sigma^2 I_n)\), known \(\sigma^2\), and slab \(\beta\mid\gamma=1\sim N(0,\tau^2)\). Under exclusion, \(\beta=0\). Let \(z=x'y/\sqrt{x'x}\) and \(r=\tau^2x'x/\sigma^2\).

Under exclusion, \(z\sim N(0,\sigma^2)\). Under inclusion, the prior uncertainty in \(\beta\) adds variance to \(z\), giving \(z\sim N(0,\sigma^2+\tau^2x'x)\). Taking the ratio of these normal densities gives the Bayes factor. This derivation assumes the single candidate predictor is orthogonal to the other regression terms after residualization; without that condition, the relevant conditional or joint marginal likelihood must be used.

The first factor penalizes the more diffuse alternative: its probability density is spread over a wider range. The exponential factor rewards an observed \(z\) that is relatively large under the null. Consequently, a larger standardized signal can favor inclusion, while a very diffuse slab can reduce evidence by spreading prior probability over effects not supported by the data.

Combining Evidence Across Specifications

With more than one candidate predictor, posterior inclusion probabilities are obtained by summing model probabilities, not by fitting each coefficient in isolation. Marginal likelihoods incorporate the fit and prior uncertainty of the entire specification. In particular, two correlated predictors may explain similar parts of the outcome, so the evidence for including one can depend on whether the other is present.

What the Inclusion Probability Does—and Does Not—Say

Under the spike-and-slab prior, \(P(\gamma_j=1\mid y)\) is the posterior probability that the model includes coefficient \(j\), under the stated likelihood and priors. Because the slab is continuous, conditional on inclusion the prior assigns probability zero to the exact value \(\beta_j=0\). The posterior for \(\beta_j\) therefore combines an atom at zero with a continuous component. This is a direct way to quantify both model uncertainty and coefficient uncertainty.

A high inclusion probability does not by itself establish that a predictor has a large effect, that its effect has a particular sign, or that a causal relationship exists. For magnitude and sign, inspect the coefficient posterior conditional on inclusion or the full mixture posterior. Causal interpretation requires a causal design and assumptions beyond this prior construction.

Practical Choices and Common Pitfalls

Prior inclusion probabilities affect selection, especially when many predictors are considered. A common choice is a shared \(\pi_j=\pi\), but this implies a prior expected number of included predictors equal to \(p\pi\). For a large candidate set, a seemingly modest \(\pi\) can still imply a substantial expected model size. Slab scales also matter: if the slab is implausibly diffuse, marginal likelihoods can favor simpler models because the alternative spreads probability over a broad range of coefficient values. Prior sensitivity analysis should therefore examine plausible inclusion probabilities and slab scales.

• Mistake: treating the most probable model as certain. Selecting the model with the largest posterior probability discards probability assigned to every other model. Report inclusion probabilities or other summaries that retain this uncertainty.
• Mistake: interpreting an inclusion probability as effect size. A coefficient can have high inclusion probability but a small conditional magnitude. Examine its conditional or mixture posterior to describe size and sign.
• Mistake: using separate one-predictor evidence when predictors are dependent. Correlated regressors can compete to explain the same variation. Use joint model evidence and sum over configurations, rather than assuming each predictor’s evidence is independent.