Why Preprocessing Must Be Part of the Resampled Pipeline
A clustering result depends not only on the clustering algorithm but also on the representation given to it. A logarithm or other transformation, a scaling rule, a feature filter, and the relative weights assigned to assay blocks can each change distances and therefore change the partition. When stability is assessed across resamples, the question is whether the analysis pipeline would produce similar subgroups under those perturbations. Fitting preprocessing once on the full data and then reusing it in every resample does not assess that pipeline cleanly.
The issue is information flow. Suppose a resample omits some patients, but the means, variances, or selected features were calculated using all patients. The omitted patients have still influenced the representation used in that resample. This is information leakage across the resampling boundary. It can make the resampled analyses look more alike than they would if each analysis had to estimate its own preprocessing from its available data. The problem can occur in unsupervised clustering even when no clinical outcome is used.
This does not mean that every operation must be newly invented in each resample. A prespecified transformation such as taking a logarithm with a fixed offset may have no estimated parameters, but any offset chosen from the observed data must be estimated anew. Likewise, a fixed rule for retaining the 500 most variable features can be prespecified, while the identities of those 500 features must be determined anew from each resample.
Formalizing the Resampled Pipeline
Let \(D\) denote the observed multi-omics data, with rows representing the resampling units and columns representing measured features in one or more blocks. Let \(I_b\) be the indices of the observations in resample \(b\). A preprocessing procedure fitted from those observations produces fitted quantities \(\hat{\theta}_b\). These may include transformation parameters, scaling constants, a retained-feature set, and block weights. The clustering algorithm then operates on the representation created using \(\hat{\theta}_b\).
The essential restriction is that \(\hat{\theta}_b\) is estimated without using observations excluded from the fitting resample. If the design requires sampling clusters of related observations, repeated measures, or patients within cohorts, the resampling unit must respect that design; preprocessing cannot repair an invalid resampling scheme. A transformation fixed in advance can be applied consistently, but any data-estimated component is refitted.
For a train-test perturbation, fit preprocessing using the training observations and apply those same fitted quantities to the test observations. This is different from refitting on the test set: independent test-set scaling would place the test data on a newly estimated coordinate system. Whether test observations can be assigned to clusters depends on whether the clustering method provides a defined assignment rule. If it does not, stability comparisons should be based on a design appropriate to that method rather than implying that held-out assignments exist.
Choose observations according to the intended bootstrap, subsampling, or train-test design and the correct sampling unit.
Estimate data-dependent transformations, scales, feature filters, and block weights using only the resample's available data.
Apply the fitted preprocessing to the resample and run the clustering procedure with its prespecified settings.
Keep the fitted choices and the resulting partition together, so changes in preprocessing can be distinguished from changes in clustering.
Worked Examples
Worked Example: Refitting a Scale
Suppose one omics feature has training-resample values 2, 4, and 6, while an excluded test observation has value 10. Use the training sample mean and sample standard deviation to standardize. Assume the feature is measured on the same scale in all four observations and has nonzero variance in the training data.
The training mean is \(\bar{x}=(2+4+6)/3=4\). The sample standard deviation is \(s=\sqrt{[(2-4)^2+(4-4)^2+(6-4)^2]/(3-1)}=\sqrt{8/2}=2\). Thus, the test value transformed using the training fit is \(z=(10-4)/2=3\). It lies three training standard deviations above the training mean.
If instead the four observations are used to fit the scale, the mean is \(5.5\), and the sample standard deviation is \(\sqrt{35/3}\approx3.416\). The test value then has standardized value \((10-5.5)/\sqrt{35/3}\approx1.317\). The omitted observation has changed the mean and standard deviation used to represent itself. In a resampling stability analysis, that is leakage if the value 10 was meant to be excluded from fitting.
Worked Example: Refitting a Variance Filter
Consider two features measured on four hypothetical patients. A resample contains the first three patients, while the fourth is excluded. The filter retains the feature with the larger sample variance. Assume both features are observed for every patient and this variance-based rule is prespecified.
| Patient | Feature 1 | Feature 2 |
|---|---|---|
| A | 0 | 0 |
| B | 1 | 2 |
| C | 2 | 4 |
| D (excluded) | 100 | 6 |
Within the resample, Feature 1 has mean 1 and sample variance \([(0-1)^2+(1-1)^2+(2-1)^2]/2=1\). Feature 2 has mean 2 and sample variance \([(0-2)^2+(2-2)^2+(4-2)^2]/2=4\). The resampled filter therefore retains Feature 2.
If the filter is incorrectly fitted using all four patients, Feature 1 has mean 25.75 and sample variance \(7352.75/3\approx2450.92\). Feature 2 has mean 3 and sample variance \(20/3\approx6.667\). The full-data filter would retain Feature 1 instead. The excluded value 100 has changed which feature enters the resampled clustering. Refitting does not guarantee the feature set remains the same across resamples; that variation is part of the pipeline's stability being assessed.
Worked Example: Refitting Block Weights
Suppose two assay blocks have already been transformed, filtered, and scaled within a particular resample. Let \(q_b\) be the mean squared Euclidean norm of a block's feature vector across the resampled patients. To give each block a target mean squared norm of 1, use the weight \(w_b=1/\sqrt{q_b}\). Assume the resample-specific calculations give \(q_A=4/3\) and \(q_B=3/4\).
For block A, the exact weight is \(w_A=\sqrt{3/4}\), so its weighted mean squared norm is \((\sqrt{3/4})^2(4/3)=(3/4)(4/3)=1\). For block B, \(w_B=\sqrt{4/3}\), and its weighted mean squared norm is \((\sqrt{4/3})^2(3/4)=(4/3)(3/4)=1\). The displayed weights are approximately 0.8660 and 1.1547, respectively. Using the rounded value for A gives \(0.8660^2(4/3)\approx0.9999\), not an exact 1; the exact equality uses the unrounded weight.
These weights must be recalculated when the resample changes, because both the scaled features and their mean squared norms can change. Equalizing these particular norms is a design choice, not a universal requirement: it gives blocks comparable aggregate scale under this definition, but does not establish that they have equal biological relevance.
Why the Refit Is Justified
A resampling analysis asks how the chosen procedure behaves when its input data are perturbed. If preprocessing is estimated from the same input that is clustered, then it is one component of that procedure. Holding its estimates fixed at values obtained from the full data instead evaluates a hybrid: a clustering step on perturbed data, conditional on preprocessing informed by observations outside the perturbation.
The distinction can be stated directly. Let \(D_{I_b}\) be the current resample and \(D_{-I_b}\) the observations left out. The intended fitted preprocessing is a function of \(D_{I_b}\) alone:
When the second expression is used inside the resample, information from \(D_{-I_b}\) affects the representation, even though those observations were meant to be unavailable for fitting. Refitting restores the stated information boundary. It does not make resamples independent, eliminate sampling variability, or prove that the resulting subgroups are valid biological entities; it makes the stability assessment correspond more closely to the specified pipeline.
Diagnostics, Alternatives, and Limitations
Keep a record for each resample of the fitted transformation parameters, scaling constants, selected features, block weights, convergence status, and cluster sizes. Inspect how often a feature is retained and whether a block receives extreme weights. Such variation can reveal a fragile preprocessing stage. Also check for failures such as zero within-resample variance, missing features, or a filter that retains too few columns. A prespecified fallback rule may be necessary, but it must be applied consistently and reported.
A useful audit is to verify that changing an excluded observation cannot alter the preprocessing fit for a resample that excludes it. In a train-test design, recompute preprocessing from training data only and check that test data are transformed with the training parameters. If an operation uses a global reference, pooled normalization, or a feature list selected before resampling, ask whether that reference was genuinely fixed independently of the study data. If not, it belongs inside the resampling loop.
| Approach | What it assesses | Main concern |
|---|---|---|
| Refit preprocessing within each resample | Stability of the specified end-to-end analysis steps under the chosen perturbation | More computation; some resamples may produce unstable or degenerate fits |
| Freeze preprocessing estimated from the full data | Clustering variation conditional on one full-data representation | Excluded observations influence the representation; not a clean assessment of the full pipeline |
| Use preprocessing fixed externally in advance | Stability conditional on that externally specified representation | Appropriate only when the reference truly is independent and the target question is conditional on it |
Refitting is not always the only defensible analysis. If the scientific target is explicitly stability conditional on a locked, externally trained transformation or feature panel, freezing that independent reference can be appropriate. The key is to define the target and information boundary before analysis. Conversely, data-adaptive normalization or feature selection performed once on the study cohort generally needs to be repeated inside each resample when the aim is end-to-end stability.
A further limitation is that refitting can make resampled pipelines heterogeneous: different feature sets or weights may be used from run to run. That is not automatically a flaw; it can expose sensitivity to preprocessing. But it means a stability result reflects both preprocessing and clustering changes. To isolate a particular source of variation, the analysis must deliberately hold other components fixed and state that narrower target. Refitting alone does not determine whether the number of clusters is appropriate or whether subgroup membership has a useful clinical interpretation.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A training resample contains values 1, 3, and 5 for one feature; a held-out value is 9. Find the training mean and sample standard deviation, then standardize 9 using those training quantities.
- A feature filter retains the top 20 variables by variance. Which observations determine the 20 selected variables in a resample, and why?
- A block has resample-specific mean squared norm \(q=2\). What weight gives it target mean squared norm 1?
- In a train-test perturbation, should the test observations be used to estimate the training scale? Explain.
- Name one diagnostic that could reveal a problematic preprocessing fit across resamples.
Solutions. (1) The training mean is \((1+3+5)/3=3\), and the sample standard deviation is \(\sqrt{[(1-3)^2+(3-3)^2+(5-3)^2]/2}=2\). Thus the standardized held-out value is \((9-3)/2=3\). (2) Only observations available in that resample determine the variances and selected 20 features; otherwise excluded observations influence the resampled representation. (3) \(w=1/\sqrt{2}\), because \(w^2q=(1/2)(2)=1\). (4) No. Estimate the scale on training observations, then apply those fitted values to test observations, so the test set does not influence fitting. (5) Examples include recording feature-selection frequencies, checking for zero-variance features, or inspecting extreme block weights and failed fits.