Bootstrap Uncertainty for Rotated Loadings

Why Bootstrap Rotated Loadings?

A rotated loading matrix is estimated from a sample, so its entries vary from sample to sample. Bootstrap resampling gives an empirical way to assess that variation: repeatedly resample the observed cases, refit the factor model, and record the rotated loadings. There is an important complication. A factor solution can describe the same fitted structure after its columns are reordered or their signs are reversed. If those equivalent versions are compared entry by entry without first aligning them, the apparent variability can be mostly a labeling artifact.

The goal here is a set of loading intervals whose coordinates refer consistently to the same factors as a chosen reference solution. The intervals describe sampling uncertainty under the chosen model, extraction method, rotation criterion, and alignment rule. They do not remove uncertainty about the number of factors or make a particular rotation uniquely correct.

The Resampling and Alignment Procedure

For independent, identically distributed observational units, each bootstrap data set contains \(n\) rows sampled with replacement from the original \(n\) rows. Each row remains a complete \(p\)-variable observation. Resampling variables separately would destroy the cross-variable dependence that the factor model is intended to describe. If observations are clustered, longitudinal, or drawn under a complex sampling design, the resampling scheme must instead respect that design.

For each resample, repeat the entire estimation procedure that produced the reference solution: use the same preprocessing, factor count, extraction method, and rotation criterion. A replicate loading matrix obtained from a different procedure is not a like-for-like estimate. Then align its columns to the reference before storing individual loading estimates.

A common alignment restricts attention to signed permutations: reorder the replicate factors and multiply selected columns by \(-1\). This handles the usual sign and factor-order indeterminacies while keeping the solution in the same rotated coordinates. More general rotational differences may require a criterion-specific admissible transformation; an arbitrary matching transformation can change the target being estimated. With oblique factors, any transformation of the loadings must be accompanied by the corresponding transformation of the factor-covariance matrix.

The equivalence follows directly. Write \(A=\hat{\Lambda}\), \(B=\hat{\Lambda}^{*}\), and let \(P\) range over signed permutation matrices. A signed permutation only reorders or changes the signs of columns, so \(\lVert BP\rVert_F^2=\lVert B\rVert_F^2\). Expanding the distance gives

For a fixed column assignment, the trace is a sum of matched column cross-products with freely chosen signs. Choosing each sign to make its cross-product nonnegative makes that sum the sum of the absolute cross-products. Comparing assignments then selects the permutation with the largest total. Thus minimizing distance and maximizing total absolute cross-product give the same signed-permutation alignment. This argument also explains why a greedy “match each column separately” rule can fail: the best overall assignment must respect the fact that each reference column is used once.

Constructing Loading Intervals

After alignment, retain the bootstrap estimate of each loading in its reference coordinate. For loading \((j,k)\), the \(b\)th aligned estimate is denoted \(\hat{\lambda}^{*(b)}_{jk,\mathrm{al}}\). The bootstrap distribution of these values can be summarized by percentiles or by centered bootstrap errors. The basic interval below makes explicit how the centered errors translate into an interval for the unknown loading.

Let \(T^{*(b)}_{jk}=\sqrt{n}(\hat{\lambda}^{*(b)}_{jk,\mathrm{al}}-\hat{\lambda}_{jk})\), and let \(c^*_{u,jk}\) be its empirical \(u\)-quantile across \(B\) replicates. A two-sided basic interval with nominal level \(1-\alpha\) is

The endpoints reverse the order of the error quantiles because the bootstrap error is subtracted from the observed estimate to obtain a candidate value for the parameter. A percentile interval instead uses the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the aligned bootstrap estimates themselves. It is simple to report, but its first-order coverage is not guaranteed for an asymmetric sampling distribution; the basic interval has a direct justification under centered bootstrap consistency.

To verify the result, conditional quantile consistency gives \(c^*_{u,jk}\xrightarrow{p}c_u\), where \(c_u\) is the \(u\)-quantile of \(F\). The event that \(\lambda_{jk}\) lies in the displayed interval is exactly

The statistic in the middle converges in distribution to \(F\), and the two random endpoints converge in probability to \(c_{\alpha/2}\) and \(c_{1-\alpha/2}\). Continuity at those endpoints therefore gives limiting coverage \(F(c_{1-\alpha/2})-F(c_{\alpha/2})=(1-\alpha/2)-(\alpha/2)=1-\alpha\). This is a first-order statement under the stated regularity and bootstrap-consistency conditions, not a guarantee for every fitted factor model. \(\square\)

Signs, Factor Order, and Oblique Solutions

The reference solution is needed because the label “factor 1” has no intrinsic meaning across independently fitted replicates. Suppose a replicate has columns in reversed order and the first column has its sign reversed. If the raw replicate is \(B=[-\hat{\lambda}_2,\hat{\lambda}_1]\), the signed permutation

gives \(BP=[\hat{\lambda}_1,\hat{\lambda}_2]\). The bootstrap records should use this aligned order and orientation before computing any loading interval. The choice of reference orientation is conventional: reversing a reference column reverses the corresponding aligned estimates and interval endpoints, but does not change the fitted covariance representation.

For oblique solutions, let \(\Phi\) denote the factor-covariance matrix. If the aligned loading matrix is \(\Lambda_{\mathrm{al}}=\Lambda P\), the corresponding covariance matrix is \(\Phi_{\mathrm{al}}=P^\mathsf{T}\Phi P\). This follows by defining the transformed factors as \(f_{\mathrm{al}}=P^\mathsf{T}f\): then \(\Lambda f=\Lambda P f_{\mathrm{al}}\), and \(\operatorname{Cov}(f_{\mathrm{al}})=P^\mathsf{T}\Phi P\). Both the loading coordinates and factor correlations must therefore be interpreted in the same aligned system.

Interpreting the Intervals and Common Pitfalls

A loading interval summarizes uncertainty in one aligned coordinate of the selected rotated solution. An interval spanning zero indicates that this loading is not precisely separated from zero under the fitted procedure; it does not, by itself, establish that the factor is absent or that a particular variable has no relationship to the broader factor structure. Intervals for separate entries are ordinarily marginal, not simultaneous: a collection of 95% intervals does not generally provide 95% joint coverage for the entire matrix.

• Do not interpret unaligned sign flips as sampling variability. A negative replicate column may represent the same factor with the opposite orientation. Align first, then summarize.
• Do not assume column positions identify factors. Match columns to the reference; factor order can change across refits.
• Do not treat any rotation as interchangeable for interval purposes. Apply the same fitting and rotation procedure, and use a defensible alignment rule that preserves the intended coordinates.
• Do not ignore failed or inadmissible fits. Record convergence and admissibility, investigate failures, and state how any excluded replicates were handled. Quietly dropping difficult replicates can distort the reported uncertainty.
• Do not read an interval as accounting for every source of model uncertainty. The procedure conditions on choices such as factor count and rotation criterion.

If aligned replicates still show very different loading patterns, the coordinate-wise intervals may be summarizing a mixture of substantively distinct solutions rather than a single regular sampling distribution. In that case, report the alignment rule and the observed variation, and be cautious about presenting a narrow interval as definitive. Bootstrap intervals are most interpretable when the chosen solution has a coherent, locally identifiable alignment across resamples.