Measuring Recovery Of Latent Structure

Why Recovery Requires Alignment

In a simulation study, the generating mechanism supplies a known latent structure, and an integration method returns an estimate of it. Directly subtracting the estimate from the truth may seem like a natural recovery measure, but it can be misleading: several numerical representations can describe the same structure. Cluster labels can be permuted, factor axes can be rotated or have their signs reversed, and factor scores and loadings can trade off in scale. A useful recovery metric first accounts for the indeterminacies that are legitimate under the model, then measures the remaining discrepancy.

The goal is not to find the alignment that makes an estimate look good under any circumstances. Alignment should remove only transformations that leave the target structure unchanged. If a method estimates the wrong number of factors, misses clusters, or selects the wrong features, no relabeling should hide that failure.

Choose the Target and Its Invariances

Let \(Z\) denote a simulated \(n \times K\) matrix of true sample scores on \(K\) latent factors, and let \(\hat{Z}\) be an estimated score matrix with the same samples and number of factors. Let \(L\) and \(\hat{L}\) denote the corresponding \(p \times K\) true and estimated loading matrices for \(p\) features. In a common factor representation, the signal is described by the product \(ZL^\mathsf{T}\). For any invertible \(K \times K\) matrix \(A\), replacing \(Z\) by \(ZA\) and \(L\) by \(L A^{-\mathsf{T}}\) leaves that product unchanged. Thus, score and loading coordinates need not be individually identifiable without additional constraints.

Many methods impose conventions such as uncorrelated factors with unit variance. Under such constraints, the remaining equivalence may be limited to sign changes and permutations, or to orthogonal rotations. The simulation's generating model and the method's output convention determine which transformations are valid. Do not allow arbitrary transformations merely because they improve a metric: they can erase meaningful differences in the factor structure.

For clustering, an assignment of observations to cluster numbers has no intrinsic order. The partitions \((1,1,2,2)\) and \((2,2,1,1)\) are identical. A cluster-recovery metric should therefore compare assignments without depending on the numeric labels. The adjusted Rand index (ARI) does this by counting pairs of observations and adjusting for the agreement expected from random assignments with the observed cluster sizes.

Aligning Factor Scores and Loadings

If estimated score axes may differ by an orthogonal rotation, orthogonal Procrustes alignment finds the rotation that minimizes the squared discrepancy between matched samples. The Frobenius norm \(\|M\|_F\) is the square root of the sum of the squared entries of matrix \(M\).

Here \(I\) is the \(K \times K\) identity matrix. If the singular value decomposition of \(\hat{Z}^\mathsf{T}Z\) is \(U\Sigma V^\mathsf{T}\), one minimizing rotation is \(\hat{Q}=UV^\mathsf{T}\). The aligned scores are \(\hat{Z}\hat{Q}\). This transformation includes sign flips and permutations as special cases, but it does not include arbitrary scaling. That is appropriate when the model's constraints make scale meaningful but leave orientation arbitrary.

The residual after alignment can be summarized by a root-mean-square error, for example \(\|\hat{Z}\hat{Q}-Z\|_F/\sqrt{nK}\). State whether scores were centered or standardized before alignment: centering removes a difference in means, while standardizing also changes scale. Either operation can conceal a real error if that difference is part of the target.

Loadings can be evaluated with related but not identical metrics. For an individual loading column, a sign-invariant directional similarity is the absolute cosine: \(|L_k^\mathsf{T}\hat{L}_k|/(\|L_k\|_2\|\hat{L}_k\|_2)\). It ranges from 0 to 1 when both vectors are nonzero, with 1 indicating the same direction up to sign. Because normalization removes magnitude, it does not measure loading size. Conversely, a raw loading RMSE is sensitive to scale and sign. The appropriate choice depends on whether the target is the direction or the numerical loading values under a defined scale convention.

Comparing Cluster Assignments

A contingency table \(n_{ij}\) records the number of observations assigned to true cluster \(i\) and estimated cluster \(j\). For any integer \(m\geq2\), \(\binom{m}{2}=m(m-1)/2\) counts pairs within a group of size \(m\). The ARI compares the number of pairs placed together in both partitions with its expected value under random agreement given the table's row and column totals.

Here \(a_i=\sum_j n_{ij}\) and \(b_j=\sum_i n_{ij}\) are the true and estimated cluster sizes, and \(n\) is the number of observations. The adjustment subtracts the expected pair agreement under the fixed-margin randomization model. An ARI of 1 indicates identical partitions; a value near 0 indicates agreement around the fixed-margin chance baseline, and a negative value indicates less pair agreement than that baseline. It is not a probability.

Justification and Choice of Metric

The Procrustes solution follows from a short optimization argument. Expanding the squared norm gives \(\|\hat{Z}Q-Z\|_F^2=\operatorname{tr}(\hat{Z}^\mathsf{T}\hat{Z})+\operatorname{tr}(Z^\mathsf{T}Z)-2\operatorname{tr}(Q^\mathsf{T}\hat{Z}^\mathsf{T}Z)\), where \(\operatorname{tr}\) is the sum of diagonal elements. The first two terms do not depend on \(Q\), so minimizing the residual is equivalent to maximizing \(\operatorname{tr}(Q^\mathsf{T}M)\), where \(M=\hat{Z}^\mathsf{T}Z=U\Sigma V^\mathsf{T}\). Set \(R=U^\mathsf{T}QV\); because \(Q\), \(U\), and \(V\) are orthogonal, \(R\) is orthogonal. Then \(\operatorname{tr}(Q^\mathsf{T}M)=\operatorname{tr}(R^\mathsf{T}\Sigma)=\sum_k \sigma_k R_{kk}\leq\sum_k\sigma_k\), since \(\sigma_k\geq0\) and \(R_{kk}\leq1\). The upper bound is attained at \(R=I\), giving \(Q=UV^\mathsf{T}\).

No single metric answers every recovery question. The comparison below helps match the measure to the structure and its legitimate indeterminacies.

For multi-factor loading matrices, align columns consistently before reporting column-wise metrics. If only the span of the factors is meaningful, principal angles are often more defensible than a column-by-column RMSE. For example, if the columns of two matrices are orthonormal bases for the same \(K\)-dimensional subspace, a rotation can make the basis vectors look different even though the subspaces coincide. Principal angles measure the orientation difference between subspaces, with all angles equal to zero for identical subspaces. Neither subspace similarity nor a high cosine for one column guarantees accurate feature selection or correct scientific interpretation.

Diagnostics, Alternatives, and Limitations

A recovery score is credible only if its alignment and preprocessing match the intended estimand. Inspect the aligned results as well as the summary metric. A good practice is to report the number of true and estimated factors or clusters, the alignment rule, and a metric appropriate to the target. In simulation replications, summarize the distribution of recovery scores rather than relying on a single run.

• Check dimensions and matching. Confirm that the same simulated samples and features are compared in the same order. Missing or excluded observations must be handled consistently; silently comparing different rows invalidates the score.
• Inspect the transformation. A very large rotation, sign flip, or scale adjustment may be allowed by the model, but should be reported. If alignment requires a transformation outside the model's invariances, it is not a fair correction.
• Check the number of recovered components. Procrustes rotation assumes compatible score dimensions. If the estimated number of factors differs from the truth, a forced one-to-one alignment can discard unmatched structure or mislead. Report dimension recovery separately, or use an explicitly defined subspace comparison that handles the dimension difference.
• Look beyond one summary. The ARI can conceal which clusters were merged or split; a loading cosine can hide poor magnitude recovery; and an aggregate score can hide one unrecovered factor. Inspect contingency tables, per-factor measures, or aligned residuals as relevant.

Alternative metrics are useful when they target a different property. For clusters, unadjusted pair agreement or information-based measures may be reported, but their chance correction and behavior under differing cluster counts are not identical to the ARI. For factors, correlation after column matching is common, while principal angles focus on the subspace. These alternatives should not be presented as interchangeable: state exactly what is invariant, what is penalized, and whether higher or lower values indicate better recovery.

A central limitation is that recovery of a simulated parameter is not the same as scientific validity. A factor may be recovered accurately because the generating model made it unusually clear; a method may also recover a chosen representation while failing to capture another scientifically relevant feature. Recovery metrics are conditional on the simulation truth, constraints, and alignment rule. They complement, but do not replace, checks that the simulated scenario is plausible.