Sparse Loading Priors For Feature Selection

Why Put Sparsity on Factor Loadings?

In a multi-omics factor model, a loading describes how a feature relates to a latent factor. If most features have little relation to a factor, a dense loading profile can be difficult to interpret: many small estimated coefficients may appear alongside the few associations that carry most of the signal. A sparse loading prior expresses the belief that many loadings are zero or near zero while allowing some to be appreciably nonzero. It therefore offers a form of model-based feature-selection evidence, not a guarantee that selected features are biologically causal.

The earlier material on Bayesian priors on scores and loadings introduced prior distributions for these model parameters. Here the focus is the additional structure that makes a loading profile sparse and the interpretation of posterior inclusion or shrinkage measures. Because factor loadings can be affected by scale, sign, and rotation, sparsity is interpreted under an identified or constrained factor representation. A sparsity prior alone does not establish that a factor has a unique biological meaning.

Spike-and-Slab Priors and Inclusion Evidence

A spike-and-slab prior introduces a binary indicator \(\gamma_{jk}\). Conditional on the factor scores and other model quantities, a simple specification is

The spike is a point mass at zero; the slab is a wider distribution that permits nonzero loadings of either sign. The prior inclusion probability \(\pi_k\) controls how often loadings for factor \(k\) are expected to be included before seeing the data. It can be fixed or given a hierarchical prior, but its choice should reflect the intended degree of sparsity and be checked for sensitivity.

The PIP combines the prior odds with the evidence in the data. For one loading, let \(H_0\) denote the point-spike model and \(H_1\) the slab model. The Bayes factor \(\mathrm{BF}_{10}\) is the marginal likelihood under \(H_1\) divided by that under \(H_0\). Posterior odds equal prior odds multiplied by this Bayes factor:

This relation makes clear why a Bayes factor favoring inclusion does not necessarily imply a high PIP: a small prior inclusion probability can leave the posterior probability modest. A PIP is conditional on the likelihood, prior, and factor representation used. It is not a frequentist p-value, nor is it the probability that a feature causes a biological outcome.

Why the Conjugate Bayes Factor Has This Form

The Bayes factor above follows by integrating the conditional likelihood over the slab prior. Under the normal observation model, the likelihood ratio for a proposed \(\lambda\) relative to \(\lambda=0\) is

Multiplying this ratio by the slab density adds the prior term \(-\lambda^2/(2\tau^2)\) to the exponent. Completing the square gives the slab posterior variance \(V=(a/\sigma^2+1/\tau^2)^{-1}\) and mean \(m=Vb/\sigma^2\). Integrating the resulting normal kernel yields the marginal likelihood ratio:

The square-root term penalizes the more flexible slab by averaging its likelihood across possible loading values. The exponential term reflects how well the data support loadings away from zero. The calculation requires the stated normal likelihood, independent residuals with common known variance in this simple derivation, and a normal slab. With unknown variance, correlated residuals, or count and binary likelihoods, the same basic comparison of marginal likelihoods applies, but this closed-form expression generally does not.

Continuous Shrinkage and Its Interpretation

A continuous shrinkage prior concentrates density near zero but assigns zero probability to the exact value \(\lambda_{jk}=0\). Examples include normal priors with feature-specific scales and global-local priors such as the horseshoe. The global scale encourages broad shrinkage across a loading profile, while local scales allow selected coefficients to escape that shrinkage. Since these priors do not have a separate point mass at zero, they do not define a PIP in the same way as a spike-and-slab model.

A useful measure in a simple normal model is the conditional shrinkage factor. Suppose \(y_i=z_i\lambda+\epsilon_i\), \(\epsilon_i\sim N(0,\sigma^2)\), and \(\lambda\sim N(0,\tau^2\psi^2)\), where \(\psi\) is a local scale. With \(a=\sum_i z_i^2\), the posterior mean is the unregularized estimate \(b/a\) multiplied by

Here \(\kappa\) near 1 indicates strong shrinkage toward zero and \(\kappa\) near 0 indicates little shrinkage, conditional on the scales and other quantities. It is a measure of shrinkage, not a probability that the loading is absent. In hierarchical models, posterior summaries of \(\kappa\) should account for uncertainty in the scales rather than reporting only a plug-in value.

Turning Posterior Summaries into Feature-Selection Evidence

Selection requires a reporting rule in addition to a prior. In a spike-and-slab model, one may report PIPs and apply a stated threshold, or retain a ranked list for follow-up. A threshold such as 0.8 is a decision convention, not a universal boundary between true and false features. The consequences of a threshold depend on the scientific costs of missed features and false leads, and on whether inclusion indicators correspond to scientifically meaningful features in an identified factor solution.

For a selected set \(S\) based on posterior PIPs, the posterior expected proportion of false inclusions, conditional on the data and the model, can be summarized by \(\sum_{(j,k)\in S}(1-\mathrm{PIP}_{jk})/|S|\), for nonempty \(S\). This is a Bayesian posterior expectation under the fitted model; it is not automatically a frequentist false discovery rate guarantee. For continuous shrinkage priors, researchers can report posterior means, credible intervals, or posterior probabilities that the absolute loading exceeds a scientifically chosen threshold. A threshold probability such as \(P(|\lambda_{jk}|>\delta\mid y)\) is not a point-mass PIP; it answers a different, effect-size-based question.

Assumptions, Diagnostics, and Alternatives

Sparse loading evidence is only as credible as its model. A conditional calculation that treats scores as fixed must justify that simplification or acknowledge it as an illustration. In a full model, uncertainty in scores, residual variances, factor structure, and hyperparameters can affect posterior inclusion. The observation model must also suit the data: continuous measurements, counts, and binary measurements generally require different likelihoods. Conditional independence or any residual covariance structure must be specified rather than assumed silently.

Useful checks include examining whether selected loadings are sensitive to plausible prior inclusion probabilities, slab scales, or global and local shrinkage choices; checking whether conclusions change under reasonable likelihood or residual-structure alternatives; and checking whether residual patterns suggest that the factor model is missing structure. For continuous shrinkage, inspect posterior loading distributions and shrinkage factors rather than relying on a single posterior mean. For spike-and-slab, report PIPs alongside effect-size summaries so readers can distinguish selection evidence from estimated magnitude. These checks diagnose modeling sensitivity; they do not by themselves prove that a selected feature is a true biological marker.

The main alternatives answer related but different questions. A dense prior permits all loadings to vary and may be suitable when broad weak effects are plausible, but can yield harder-to-read profiles. Spike-and-slab gives explicit point-null inclusion probabilities, at the cost of discrete indicators and sensitivity to inclusion and slab priors. Continuous shrinkage avoids discrete inclusion variables and can strongly compress weak coefficients while preserving some large ones, but exact-zero selection needs an additional reporting rule. Penalized methods such as the lasso can produce exact zeros as optimization estimates, but those zeros are not posterior inclusion probabilities. None of these choices removes factor rotation or confounding issues; sparse structure must be interpreted within the factor model’s identification constraints.

Common Pitfalls and Limitations

• Calling a PIP a probability that the null is true. A PIP is the posterior probability of the specified inclusion indicator, conditional on the model and prior. It is not a universal probability that a feature is biologically irrelevant.
• Equating shrinkage with selection. A continuous shrinkage factor quantifies contraction in a particular parameterization; it is not the probability of a point-null state.
• Reading a posterior mean as inclusion evidence. A small mean may result from a low PIP, a small included effect, or positive and negative uncertainty. Report the relevant posterior summaries separately.
• Ignoring prior sensitivity or factor identification. Different plausible prior scales can change PIPs, and factor sign, scale, or rotation ambiguity can change which loading is being summarized.

Sparse priors are useful when concise loading profiles support scientific interpretation and follow-up. Their limitation is that sparsity is partly a modeling assumption: a prior favoring few nonzero loadings can understate diffuse weak effects, while a broad slab can make inclusion evidence sensitive to its scale. In multi-omics work, selection should therefore be reported as conditional evidence under a stated model, with uncertainty and sensitivity made visible.