Stability Of Subject Scores Under Resampling

Why Subject-Score Stability Matters

A fitted factor model assigns each subject a score on each factor. A score can place a subject high or low relative to others, but that position is estimated from data and may change when the subjects used to fit the model change. A clear-looking score from one fit is not, by itself, evidence that the subject’s position is reproducible.

This tutorial assesses subject-score stability under resampling: whether subjects retain similar scores, ranks, and relative ordering across fits to perturbed samples of subjects. The emphasis is on the subjects, not on re-evaluating which features define a factor. Stability is always relative to a specified factor, score scale, resampling scheme, and set of subjects being compared.

Set Up a Meaningful Comparison

Resampling changes the data used to estimate a factor model. Therefore, compare scores only after making their meanings comparable across fits. Factor labels can switch, and a factor’s sign can reverse without changing the underlying solution. Use the factor-matching and sign-alignment procedures introduced earlier in the course; apply the corresponding transformations to the scores as well as to the loadings. If a model allows more general rotations, compare coordinates only after a suitable common orientation has been established.

Scores also need a common scale. Use a model parameterization that fixes the score scale, or express scores in a consistent reference scale across fits. A numerical change caused only by one fit using a different scale is not evidence that a subject moved biologically. Do not choose an alignment or scale transformation to make the subjects’ rankings agree: that would build the desired stability into the comparison.

A direct strategy is to define a fixed set of anchor subjects and score them under every resampled fit. One way is to resample subjects to fit the model, then estimate the anchor subjects’ scores under that fitted model using the same scoring rule each time. Anchors not included in a particular training resample can be scored out of sample, when the model supports this. Scoring a fixed set makes the across-fit comparison explicit; using only the subjects present in each resample can leave different fits with different comparison sets.

For clustered or repeated-measures data, resample the independent sampling units or clusters rather than treating dependent rows as independent. For stratified sampling, preserve the strata. The resampling scheme should mimic how new data would vary under the target design; otherwise the resulting stability summaries answer the wrong question.

Summaries of Score and Rank Stability

For a fixed subject \(i\) and factor \(k\), calculate the resampling mean and sample standard deviation:

These summarize the center and spread of that subject’s scores across the specified resampling procedure, in the chosen score units. They are not automatically a standard error or a confidence interval for a biological quantity. Under suitable sampling and regularity conditions, bootstrap quantiles can approximate sampling uncertainty; that interpretation still depends on a valid model, resampling design, and scoring procedure.

For rankings, define rank 1 as the highest score and use midranks for ties. A useful summary is the rank correlation between a reference fit’s ranks and the ranks from each resampled fit. Spearman’s rank correlation measures similarity in ordering; it does not require score differences to be linear or equally spaced. Also report summaries that answer a concrete question, such as how often a subject ranks first or how often it falls in the top 10%. These are proportions across the resampled fits, not probabilities that a subject has a true rank.

For two subjects \(i\) and \(j\), the empirical pairwise ordering frequency is

where \(I(\cdot)\) equals 1 when the stated ordering holds and 0 otherwise. If ties matter, state how they are handled rather than silently counting them as one direction. A value near 1 indicates that \(i\) tends to score above \(j\) in these resampled fits; a value near 0.5 indicates that their ordering often changes.

Why Spearman’s Rank Summary Has This Form

When there are no tied ranks, let \(r_i\) and \(s_i\) be the ranks of subject \(i\) in two fits, and let \(d_i=r_i-s_i\). Both rank lists are permutations of \(1,\ldots,n\), so each has mean \((n+1)/2\) and centered sum of squares \(S=n(n^2-1)/12\). The identity

follows by expanding the square; the centered sums are unchanged because both rank lists have the same mean. The Pearson correlation of the two rank lists is their centered cross-product divided by \(S\). Substituting the identity, with both centered sums equal to \(S\), gives

Thus identical ranks give \(\rho_S=1\), while larger squared rank changes reduce the correlation. With tied ranks, calculate the Pearson correlation of the midranks directly; the displayed shortcut need not apply.

Diagnostics, Comparisons, and Limits

Before interpreting a stability summary, check that every fit converged, that the same factor was matched across fits, and that the scores use a common scale. Examine the distribution of scores and ranks across fits rather than relying only on an average. A few extreme fits may inflate the score standard deviation; repeated rank reversals may be hidden by an apparently reasonable mean score. If a fit cannot be matched unambiguously, report that ambiguity rather than forcing a correspondence.

The main summaries answer different questions. Rank correlation assesses overall ordering; top-rank frequency assesses persistence at an extreme; score standard deviation measures variability in the score scale; and pairwise ordering frequency focuses on a particular comparison. None alone establishes that a factor is biologically meaningful. Loading-profile stability concerns which features define a factor, whereas subject-score stability concerns where subjects lie on a matched factor. Strong loading stability can coexist with unstable subject positions, for example when subjects have similar scores.

A posterior distribution for scores is an alternative uncertainty summary when the fitted model provides one. It represents uncertainty conditional on that model and its priors; resampling instead examines variation under repeated perturbations of the sampled subjects and refitting procedure. Neither approach removes model misspecification, confounding, measurement error, or an inappropriate sampling design. A single-fit score or model-based standard error may be simpler, but does not by itself reveal whether refitting on changed subjects alters rankings.

The resampling conclusions are conditional on what was held fixed. If features, preprocessing, priors, or model complexity are also changed, score instability mixes those sources with subject-sample variation. Conversely, fixing them means this assessment cannot establish robustness to those other choices. A large resampling spread also does not show that any one subject is incorrectly measured; it shows that the estimated position is sensitive to the specified resampling process.