Resampling To Assess Module Stability

Why Assess Module Stability?

A co-expression module is an estimated grouping of genes, not a directly observed biological object. Its membership depends on the expression samples and on choices such as the association measure, network transformation, clustering method, and tree-cut parameters. A module that appears in one fitted network may change substantially when a few biological samples are omitted or replaced. Resampling asks whether the estimated structure is reproducible under specified perturbations of the observed samples.

The aim is not to prove that a module is biologically real. Stability describes sensitivity to the sampling variation and analysis pipeline being examined. A stable module may still reflect an unmeasured technical or biological factor; an unstable module may contain genuine relationships that the available sample size cannot estimate reliably. The earlier material on batch, cell mixture, and spurious co-expression is especially relevant: resampling does not remove confounding that is present throughout the data.

A Comparable Resampling Workflow

Let the expression data contain \(n\) biological samples measured on a fixed set of genes. Fit the chosen network and module-detection procedure to the original data to obtain reference modules \(M_1,\ldots,M_K\). For each resample, perturb the biological samples, rebuild the network, and detect modules using the same defined procedure. Rebuilding matters: resampling only the final module labels would not assess whether the network and clustering steps reproduce those labels.

For independent, similarly sampled biological units, a common option is the nonparametric bootstrap: draw \(n\) sample indices with replacement from the \(n\) observed samples. Another option is subsampling without replacement, often using fewer than \(n\) samples per refit. These perturbations answer related but different questions. Bootstrap refits retain nominal sample size but contain duplicates; subsamples deliberately fit to smaller data sets. The choice should be stated rather than treated as an interchangeable technical detail.

The independent sampling unit must match the study design. If measurements are paired, repeated within a subject, or clustered by donor, resample the appropriate independent units or intact clusters rather than individual rows. If samples come from distinct strata or conditions, the resampling scheme should preserve the design relevant to the question. Otherwise, changes in module membership may reflect a changed sample composition rather than ordinary sampling variability.

For a fair comparison, keep the gene universe and analysis rules explicit. Use the same association definition, network transformation, clustering procedure, and module-cutting rules across refits unless the purpose is specifically to examine sensitivity to those choices. Refit steps that depend on the resampled data as part of the pipeline. Control stochastic clustering by fixing its random seed or repeating the algorithm enough to distinguish algorithmic randomness from sample-resampling variation. A module absent from a replicate may be genuinely unresolved there; do not silently drop it from summaries.

Matching Modules and Measuring Overlap

Module labels such as “1” and “2” have no meaning across refits. Compare gene sets, not labels. For a reference module \(M_k\) and a resampled module \(M^*_l\), their Jaccard overlap is the size of their intersection divided by the size of their union:

The score is between 0 and 1. It equals 1 for identical gene sets and 0 for disjoint sets. Because the union includes genes found in either set, Jaccard overlap penalizes both missing reference genes and extra resampled genes. A directional retention measure answers a different question: the fraction of a reference module retained in a candidate resampled module, \(|M_k\cap M^*_l|/|M_k|\). Reporting both can clarify whether a low Jaccard score reflects loss from the reference module, additions to it, or both.

To give each reference module at most one counterpart in a replicate, match modules by maximizing total overlap across the module pairs. This is a one-to-one assignment problem; it avoids matching several reference modules to the same large resampled module and then describing all of them as reproduced. If a matched overlap is too small to be meaningful, classify the module as unmatched using a prespecified rule. There is no universal cutoff: the threshold depends on module sizes, purpose, and desired stringency, so report it and show the underlying overlap values.

Assignment Consistency Across Resamples

Module-level overlap can hide genes that repeatedly move between groups. A complementary summary tracks each gene’s assignments. For a gene \(g\) in a reference module, define \(I_{gb}=1\) if, in resample \(b\), it is assigned to the matched counterpart of that reference module, and \(I_{gb}=0\) otherwise. An unassigned gene or a gene assigned elsewhere counts as 0. Across \(B\) resamples, its assignment consistency is:

This is the observed fraction of resampled fits in which the gene retains its reference-module assignment. It is a descriptive frequency under the chosen resampling scheme, not the probability that the gene truly belongs to that module. The module-level mean of \(C_g\) can summarize assignment consistency, but the distribution across genes is informative: a high average can conceal a subset of genes with low consistency.

A related pairwise summary is the co-assignment frequency. For genes \(g\) and \(h\), calculate the fraction of resamples in which both are assigned to the same module. This can reveal a stable core when a broader module splits variably. State how unassigned genes are handled; a common choice is not to count two unassigned genes as co-assigned, since that would imply agreement without a detected module.

What the Resampling Summary Can Establish

A module-level stability profile can include, for each reference module, its matched Jaccard overlap in every resample, the fraction of resamples in which it has an acceptable match, its size, and gene-level assignment consistency. A compact average is useful, but report the spread and the frequency of missing or split matches as well. For example, an average overlap of 0.65 could arise from consistently moderate overlaps or from a mixture of near-perfect matches and failures; those patterns have different implications.

The bootstrap resamples observed biological units; it does not generate new biological diversity beyond what those units represent. Under independent, identically distributed sampling, each observed sample has probability \((1-1/n)^n\) of being absent from an \(n\)-draw bootstrap resample. As \(n\) grows, this approaches \(e^{-1}\), so a bootstrap replicate contains about 63.2% of the distinct observed samples on average. This helps explain why duplicates do not mean that a replicate contains \(n\) independent observations.

A short justification for assignment consistency is direct. Each \(I_{gb}\) is an indicator that the specified event occurred in resample \(b\). Summing these indicators counts the successful assignments, and dividing by the total number \(B\) gives the empirical relative frequency. Increasing \(B\) reduces Monte Carlo noise in this frequency, but it does not fix biased sampling, confounding, a poor network estimator, or a too-small number of biological samples.

Diagnostics, Alternatives, and Limitations

Interpret stability summaries alongside diagnostics. Check whether sample composition, batch, or cell mixture changes markedly across resamples; whether a few influential samples drive the network; whether modules frequently merge, split, or become unassigned; and whether results depend strongly on module size. Very small modules can have coarse, volatile overlap values, while a large module can achieve a seemingly respectable overlap despite substantial uncertainty in its boundaries. Check that each replicate used the same gene universe and that labels were matched by overlap rather than their arbitrary numbers.

Resampling is not the only way to examine robustness. Repeating a fit with a different network threshold, correlation measure, or tree-cut setting is a sensitivity analysis of modeling choices, not sample resampling. Independent validation data assess reproduction in another sample, with their own design and comparability concerns. Repeated algorithm runs with fixed data assess computational randomness. These approaches answer distinct questions; a strong analysis does not describe one as a substitute for the others.

The main limitation is scope: high stability means reproducibility under the specified perturbation and pipeline, not biological validity, causal relevance, or transportability to another cohort. Low stability may reflect limited sample information, an inherently continuous structure that a hard partition simplifies, or genuine heterogeneity. Stability summaries should therefore inform how confidently module membership is described, not be treated as a pass/fail test of biological truth.