Bayesian Priors On Scores And Loadings

Why Put Priors on Factor Model Parameters?

A factor model represents each sample’s measurements through a smaller set of latent scores and feature loadings, plus residual variation. The likelihood describes how the observations relate to those quantities; priors describe plausible parameter values before observing the data. In a multi-omics analysis, priors can stabilize estimates when the number of features is large relative to the number of samples, encode a chosen scale, and express uncertainty rather than returning only a single fitted solution.

A prior is not a substitute for an appropriate likelihood or an identifiable parameterization. The earlier material on rotational invariance and scale constraints is important here: different combinations of scores and loadings can represent the same fitted signal. Isotropic normal priors can be invariant to orthogonal rotations, so they do not by themselves identify a unique orientation for the factors. Any constraints or conventions used to interpret factors must be stated separately.

Model and Common Prior Choices

For continuous measurements, let \(y_{ijv}\) denote the measurement for sample \(i\), feature \(j\), and omics view \(v\). A simple conditionally Gaussian factor model is

Here \(z_{ik}\) is the score of sample \(i\) on factor \(k\), \(\lambda_{jkv}\) is the loading of feature \(j\) in view \(v\) on that factor, and \(\psi_{jv}\) is the residual variance for that feature and view. This specification assumes conditional independence of residuals across observations and features given the scores and loadings, and Gaussian residuals with mean zero. A common simplifying model instead uses one residual variance per view. These choices are modeling assumptions, not consequences of factor analysis.

The standard-normal score prior sets the factor-score scale and centers scores at zero. The loading prior is centered at zero and gives a typical loading magnitude set by \(\tau_v\). If measurements have been standardized, \(\tau_v\) is interpreted on that scale; if not, it must be chosen in the units of the measurements per unit score. A smaller \(\tau_v\) pulls loadings more strongly toward zero. The inverse-gamma prior is a convenient positive prior for a variance and yields a simple conditional update under Gaussian residuals. Its parameters \(a_v\) and \(b_v\) jointly determine its shape and scale; neither should be selected without considering the units and plausible residual magnitudes.

These priors are often chosen to make conditional posterior calculations tractable. Independence is a convenient baseline, not a universal scientific truth. For example, the same loading prior variance across features assumes comparable loading scales after preprocessing. If views have different measurement scales, view-specific prior scales or suitable preprocessing may be needed. A normal loading prior is generally a continuous shrinkage prior: it favors small values but assigns probability zero to an exactly zero loading. It is not a feature-selection prior.

How Normal Priors Produce Shrinkage

Consider one loading \(\lambda\), with known scores \(z_i\), a known residual variance \(\psi\), and observations \(y_i=z_i\lambda+\epsilon_i\). Suppose \(\epsilon_i\) are independent \(N(0,\psi)\) and the prior is \(\lambda\sim N(0,\tau^2)\). Multiplying the Gaussian likelihood by the normal prior gives a normal posterior. Completing the square in \(\lambda\) gives the posterior precision as prior precision plus data precision:

This expression makes the influence of the prior explicit. The posterior mean is a precision-weighted compromise between the prior mean, zero, and the data-supported estimate. When the data precision \(\sum_i z_i^2/\psi\) is large relative to \(1/\tau^2\), the posterior is driven more by the data. When it is small, the mean is pulled toward zero and the posterior variance is closer to the prior variance.

Scores Also Have Posterior Distributions

A prior on scores makes uncertainty about sample positions in latent space explicit. For one sample with \(K=1\), suppose measurements from \(J\) features follow \(y_j=z\lambda_j+\epsilon_j\), with independent \(\epsilon_j\sim N(0,\psi_j)\), known loadings, and \(z\sim N(0,1)\). Completing the square gives

The score posterior mean balances the prior value zero against evidence across features. Features with larger residual variance contribute less, while features whose loadings are informative relative to their residual noise contribute more. This is a conditional update: in a full Bayesian analysis, scores, loadings, and residual variances are uncertain jointly, and posterior computation must account for that uncertainty.

Residual Variance Priors and Their Update

For a feature with \(n\) independent Gaussian residuals \(e_1,\ldots,e_n\), suppose \(e_i\mid\psi\sim N(0,\psi)\) and \(\psi\sim\operatorname{IG}(a,b)\). The likelihood contributes a factor \(\psi^{-n/2}\exp[-\sum_i e_i^2/(2\psi)]\). Multiplication by the inverse-gamma prior therefore gives

The posterior shape increases with the number of residual observations, and the scale increases with their sum of squares. This update assumes that the residuals are conditionally independent, have the stated common variance, and are centered at zero under the model. Residuals in a fitted factor model are not directly observed as fixed quantities; in a Bayesian analysis, this conditional update is used alongside uncertainty in scores and loadings rather than pretending those quantities are known.

Prior Scale, Diagnostics, and Alternatives

Prior scales affect both posterior location and spread. A narrower loading prior usually produces stronger shrinkage toward zero and a smaller posterior variance when data are limited. A broader prior allows larger loadings and typically leaves more uncertainty. The same principle applies to score priors: changing their variance changes the regularization of sample scores and, together with loading scales, changes the model’s effective signal scale. For residual variance priors, changes in \(a\) and \(b\) affect the posterior, especially when there are few residual observations.

Check the implications of the priors before interpreting fitted factors. Useful checks include whether prior draws generate measurements in plausible ranges, whether their implied cross-feature or cross-view covariance is plausible, and whether conclusions change materially under reasonable alternative scales. Inspect posterior distributions for parameters pressed against regions where the prior has little support. Strong posterior sensitivity to modest prior changes is a warning that the data may not identify the parameter precisely. A prior predictive check draws parameters from the prior, then data from the likelihood, and asks whether the simulated data resemble plausible measurements.

A normal loading prior is a simple alternative to leaving loadings unregularized, as in maximum-likelihood fitting. Maximum likelihood can be useful for point estimation, but it does not by itself provide a posterior distribution or encode prior information. Variational inference approximates a posterior with a chosen family of distributions and can be computationally efficient; its uncertainty estimates depend on that approximation. A full Bayesian computation targets the posterior more directly but can be computationally demanding and remains sensitive to model and prior choices. None of these computational approaches repairs a misspecified likelihood or resolves rotational ambiguity automatically.

The assumptions and limitations should remain visible. Gaussian observation models may be unsuitable for counts or binary measurements, which require likelihoods appropriate to those data. Independent feature-level residual variances may fail when residual dependence remains. Independent normal priors do not encode known pathways, groups, or exact sparsity. Moreover, posterior summaries for individual factors can be hard to compare when sign, order, or rotation are not fixed: the fitted signal may be stable while a particular coordinate representation is not. Prior choice should therefore complement, not replace, explicit identification conventions and substantive checks.