Why Factor Axes Can Be Ambiguous
A factor model summarizes many measured features using a smaller number of latent directions. In multi-omics data, these directions may describe coordinated patterns across several assay views. But a fitted reconstruction does not always determine one unique set of axes: the same reconstruction can be expressed using rotated scores and loadings. The numerical coordinates assigned to each factor may therefore depend on an orientation choice, even when the model’s fitted values do not change.
This is not merely a cosmetic property. If two mathematically equivalent fits assign different loading patterns to “Factor 1” and “Factor 2,” interpreting either axis as a uniquely identified biological signature requires care. The central question is which transformations preserve the model, and under what assumptions.
Reconstruction Is Unchanged
The score matrix has one row per sample and one column per factor; the loading matrix has one row per feature and one column per factor. Thus, \(ZB^T\) is \(n\times p\), matching the data matrix. The same factor-coordinate transformation must be applied to both matrices for their product to remain unchanged.
The middle equality uses \((BQ)^T=Q^TB^T\), and the final equality uses \(QQ^T=I_K\). Thus every entry of the fitted data matrix is identical before and after rotation. If an intercept or other term is added to the reconstruction, it is also unchanged provided that term is left fixed.
Worked Example: Rotating a Two-Factor Reconstruction
Suppose three hypothetical features are measured on two samples, with score and loading matrices
The reconstructed values are
Use the hypothetical rotation
Its columns have squared length \(1\) and dot product \(0\), so \(Q^TQ=I_2\). Applying it gives
For the first sample, the three rotated score-loading products are \(3/2-1/2=1\), \(3/2+1/2=2\), and \(3\). For the second sample they are \(1/2+3/2=2\), \(1/2-3/2=-1\), and \(1\). The rotated matrices therefore reproduce exactly the same \(2\times3\) reconstruction.
When Does the Likelihood Also Stay the Same?
Preserving a fitted mean preserves the likelihood when the observation model and its other parameters are unchanged and the likelihood depends on the factors through that mean, or through a linear predictor constructed from it. For example, in a Gaussian factor model with fixed residual covariance \(\Psi\), each sample has conditional distribution \(Y_i\mid z_i\sim N(Bz_i,\Psi)\), using column-vector notation for \(Y_i\), \(z_i\), and the corresponding loading matrix. Because \(Z^* (B^*)^T=ZB^T\), all conditional means are the same; the residual covariance is also the same. Each observation consequently has the same density and the joint conditional likelihood is identical.
The argument also applies to observation models whose likelihood depends on the factors through an unchanged linear predictor, such as a generalized linear observation model with predictor \(ZB^T\). The likelihood equality does not follow merely from the word “factor”: it depends on how the model uses the scores and loadings. Additional terms, constraints, or penalties can distinguish orientations.
There is a second, common setting in which the likelihood is marginal over latent scores. Suppose the score vectors are independent with \(z_i\sim N(0,I_K)\). Under the rotation, the column-vector score becomes \(z_i^*=Q^Tz_i\). Since
the rotated score has the same standard multivariate normal distribution. At the same time, \(B^*z_i^*=BQ Q^Tz_i=Bz_i\). Integrating over the scores therefore gives the same marginal distribution for the observations and the same marginal likelihood. This argument assumes an isotropic standard normal score distribution; a non-isotropic prior generally does not remain unchanged under every orthogonal rotation.
Worked Example: Checking a Gaussian Conditional Likelihood
Consider two hypothetical observations of one feature. Let \(Z=I_2\), let the two loadings be \(B=(1,2)^T\), and assume independent Gaussian residuals with variance \(\sigma^2=1.25\). The observed values are \(y_1=1.5\) and \(y_2=1.0\). The fitted means are \(1\) and \(2\), so the residuals are \(0.5\) and \(-1\). Their squared sum is \(0.5^2+(-1)^2=1.25\).
For independent Gaussian residuals, the conditional log-likelihood is
Substituting \(n=2\), \(\sigma^2=1.25\), and the squared-residual sum gives
For any orthogonal \(2\times2\) matrix \(Q\), replacing \(Z\) by \(ZQ\) and \(B\) by \(BQ\) leaves their product unchanged. The fitted means remain \(1\) and \(2\), so the residuals, squared-residual sum, and log-likelihood remain exactly as calculated. This conclusion checks the model conditions: the residual variance is held fixed, and the Gaussian likelihood depends on the factors through the fitted means.
A Multi-Omics Covariance Example
The same ambiguity is visible in the implied covariance of a Gaussian factor model. If \(z_i\sim N(0,I_K)\), \(Y_i=Bz_i+\epsilon_i\), and \(\epsilon_i\sim N(0,\Psi)\) independently of \(z_i\), then \(\operatorname{Cov}(Y_i)=BB^T+\Psi\). Rotation preserves the factor contribution because \((BQ)(BQ)^T=BQQ^TB^T=BB^T\).
Worked Example: Preserving Cross-Omics Covariance
Imagine two hypothetical features, one from each of two assay views. Let their two-factor loading matrix and residual covariance be
The factor covariance and total covariance are
Use the same \(45\)-degree rotation as in the previous example. Then
The factor variance of the first feature is \(1/2+1/2=1\); that of the second is \(9/2+1/2=5\). Their factor covariance is \(3/2-1/2=1\). Adding the same residual variances again gives the same total covariance matrix, including the cross-view covariance of \(1\).
Why the Axes May Lack a Unique Orientation
A rotation changes coordinates within the latent space without changing the reconstructed data. It can redistribute a feature’s loading across factors, and can change the pattern of scores assigned to each factor. Consequently, “Factor 1” is not automatically a uniquely defined direction just because an estimation algorithm returned a first column. The fit may identify a latent subspace or a covariance structure without identifying a preferred set of axes within that space.
Some transformations are easy to recognize as special cases of orthogonal rotations: changing the sign of an axis or reordering axes. More general rotations can blend several axes. These are relevant when interpreting loading patterns, comparing fitted solutions, or reporting factor-specific results. In contrast, the reconstructed data and quantities such as \(BB^T\) remain invariant under the rotation considered here.
Rotational invariance can be broken by model structure. For example, a restriction that some loadings must equal zero, a penalty that treats particular loading patterns differently, or a prior that is not rotationally symmetric can make some orientations permissible and others not. Likewise, in a model with shared and view-specific factors, an arbitrary rotation that mixes those designated groups may violate the model’s structure. The invariance proof applies only when the transformation stays within the allowed factor space and all model assumptions remain satisfied.
Diagnostics, Alternatives, and Limitations
To diagnose orientation ambiguity, fit or inspect equivalent solutions under different initializations or rotations. Compare invariant quantities—such as the reconstructed matrix, fitted covariance, or likelihood—alongside the factor-specific scores and loadings. If the invariant quantities agree while individual columns differ, the difference may reflect orientation rather than a different fit. If likelihoods or reconstructions differ, the solutions are not equivalent under the conditions established above; convergence problems, constraints, or other model terms may be involved.
A common mistake is to assume that the largest loading in a returned column proves that the corresponding factor axis is uniquely determined. It does not: another orientation can preserve the fit while redistributing those loadings. A correction is to distinguish the fitted subspace and invariant predictions from the chosen coordinate representation, and to state any convention or constraint used to select an orientation.
Several strategies address the ambiguity, but they answer different questions. One may impose identifying restrictions as part of the model, or choose a post-fit orientation using a criterion that favors a desired loading pattern. A post-fit criterion selects a convenient representation; unless it is built into the statistical model, it does not make that orientation uniquely determined by the original likelihood. Oblique rotations, which allow non-orthogonal transformations, can be useful in other settings, but they do not generally preserve the identity score covariance or the orthogonal invariance proved here. The choice of orientation also cannot by itself establish that a factor corresponds to a causal mechanism or a distinct biological process.
The result is limited to transformations and assumptions that preserve the full model. It does not imply that every arbitrary change of scores and loadings leaves a likelihood unchanged, nor that the same rotation remains permissible under all constraints. In particular, the proof relies on orthogonality for the product identity and, for the standard normal latent-score argument, on rotational symmetry of the score distribution.
Confirm that \(Z\) is \(n\times K\), \(B\) is \(p\times K\), and the proposed \(Q\) is \(K\times K\) with \(Q^TQ=I_K\).
Calculate \(Z^*(B^*)^T\) and check it equals \(ZB^T\). For covariance models, also check \((BQ)(BQ)^T=BB^T\).
Determine whether the observation model depends on factors only through the unchanged product, and whether the latent-score distribution and any constraints are preserved.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For \(Z^*=ZQ\) and \(B^*=BQ\), which matrix identity gives the unchanged reconstruction?
- Why does a standard normal latent score distribution remain unchanged under an orthogonal rotation?
- A fitted model has the same likelihood after rotation, but different columns of loadings. What can be concluded about factor-specific interpretation?
- Give one condition under which the rotational likelihood argument may fail.
Solutions. (1) Since \(Q^TQ=QQ^T=I_K\), \((ZQ)(BQ)^T=ZQQ^TB^T=ZB^T\). (2) For \(z\sim N(0,I_K)\), the rotated score \(Q^Tz\) has mean zero and covariance \(Q^TI_KQ=I_K\), so it has the same distribution. (3) The loading columns are not uniquely determined by the unchanged likelihood alone; the difference may be a change of orientation, so factor-specific claims should account for the chosen orientation. (4) Examples include a non-rotationally symmetric score prior, a constraint or penalty that changes under rotation, or a rotation that mixes factor groups in a structured model.