Why Null Scenarios Need Careful Construction
A method can return an apparently convincing integrated pattern even when the target signal is absent. It may exploit chance correlations, batch structure, shared covariates, or an implementation leak rather than the biological relationship the benchmark is intended to assess. Null scenarios and negative controls make these behaviors visible: they provide cases in which a specified discovery should not occur.
“Null” must be defined relative to a target. A scenario with no shared cross-omics signal may still have strong structure within each omics block. A scenario with no association between omics and an outcome may still have realistic outcome prevalence and covariate effects. A shuffled cross-block control may retain each block’s internal patterns but break which samples correspond across blocks. These are different tests, not interchangeable versions of one universal null.
State the target before constructing the data. For example, “no shared signal” may mean that the latent factors generating one omics block are independent of those generating another; it does not necessarily mean every pair of measured features has zero sample correlation. “No outcome association” may mean no association conditional on specified covariates, which is different from unconditional independence. A control is interpretable only if its null claim is precise.
Three Targets, Three Construction Strategies
A global no-shared-signal scenario removes shared latent drivers across modalities. It can retain modality-specific factors, feature correlations within each modality, marginal distributions, and cohort structure. In a simple generator, use separate latent variables for the two blocks and draw them independently. If a covariate such as batch drives both blocks, however, the observed blocks can still be associated marginally. Decide whether the target is absence of biological shared signal or absence of all cross-block association, and represent nuisance structure accordingly.
A no-outcome-association scenario removes the association between the predictors under evaluation and the outcome. The outcome can still depend on covariates that are not the target predictors. If the intended null is conditional on those covariates, the generator should preserve their effects and remove only the predictor effect. Generating the outcome independently of everything instead would test a stronger and potentially unrealistic null.
A shuffled cross-block control keeps the observed values in each block but changes the sample correspondence between blocks. When samples are exchangeable, randomly permuting the rows of one block relative to the other preserves that block’s empirical distribution and within-block covariance while disrupting its original pairing. If samples are clustered, repeated, or stratified by important covariates, unrestricted shuffling may create implausible pairings or leave confounding intact. Use permutations within appropriate exchangeability groups, or generate data conditionally on the relevant structure.
Worked Examples
Worked Example: A No-Shared-Signal Benchmark
Suppose a hypothetical benchmark has four matched samples and two one-feature omics blocks. The generator draws the block-specific latent values independently. One illustrative realization is \(X=(-1,-1,1,1)\) and \(Y=(-1,1,-1,1)\). Both vectors have sample mean zero. Their sample covariance is
The sample variances are \(s_X^2=s_Y^2=4/3\), so the sample correlation is \(0/\sqrt{(4/3)(4/3)}=0\). The blocks retain nonzero variation, while this realization has no linear cross-block association. In a real simulation, independent draws will generally produce nonzero sample correlations by chance; the defining null is independence in the generator, not a requirement that every simulated data set have exactly zero sample correlation.
Suppose a method reports at least one shared-signal finding in 6 of 100 independently generated benchmark replicates. Under this global null, each such finding is false for the target. The empirical probability of at least one false finding is \(6/100=0.06\), or 6%. This estimate describes the simulated replicates; it is not proof that the method’s error rate is exactly 6% in every population or setting.
Worked Example: Removing an Outcome Association
Consider a hypothetical complete-null scenario with predictor \(X=(1,2,3,4)\) and an outcome vector independently generated and paired as \(Y=(2,4,1,3)\). The means are \(\bar{x}=2.5\) and \(\bar{y}=2.5\). The sample covariance is
Here \(s_X^2=s_Y^2=5/3\), so the sample correlation is also zero. This arithmetic describes the displayed realization, not a condition that an independent outcome generator must enforce. Under a continuous generator, an exactly zero sample correlation is unusual; random finite samples fluctuate around the population null.
If the intended null were conditional on a covariate \(C\), the scenario would need to retain the relationship between \(C\) and \(Y\) while removing the target association between \(X\) and \(Y\) given \(C\). For example, generate \(Y\) from a model involving \(C\) but omit the \(X\) term. Alternatively, a permutation control could rearrange outcomes only within exchangeable groups defined by \(C\). A global shuffle would not generally preserve the conditional structure.
Worked Example: Shuffling Cross-Block Correspondence
Suppose a hypothetical first block has values \(X=(1,2,3,4)\), and the second block originally has the same values in the same sample order, \(Y=(1,2,3,4)\). The original sample correlation is 1. Now permute the second block’s rows using the order \(2,4,1,3\), giving \(Y^*=(2,4,1,3)\). Its mean remains 2.5, and
Since both sample variances are \(5/3\), the shuffled sample correlation is 0. The permutation preserves the second block’s four values and the first block’s values, but breaks their original correspondence. With multiple features in the second block, applying the same row permutation to the whole block preserves its within-block sample covariance matrix: the observations are reordered together, so all pairwise feature products remain the same. The control does not guarantee zero cross-block association in every random permutation; chance alignment can remain.
What Permutation Preserves—and What It Does Not
The rationale for a shuffled control can be stated directly. Let \(x_i\) and \(y_i\) be centered values for two features, and let \(\pi\) be a uniformly sampled permutation of \(1,\ldots,n\). The shuffled cross-product is \(\sum_i x_i y_{\pi(i)}\). Because each \(y_j\) is equally likely to occupy each position, the expected shuffled value at any position is the mean of the \(y\) values, which is zero. Therefore,
This is an expectation over random permutations, not a guarantee for one permutation. It also relies on centering and on the validity of treating the rows as exchangeable. With strata, a within-stratum permutation has the corresponding zero-mean property for centered values within each stratum, provided the permutation is uniform within those groups. If the rows are not exchangeable, the calculation does not justify unrestricted shuffling.
Permutation is only one option. Independent generation gives a direct model-based null, but can erase realistic marginal and within-block structure if the generators are too simple. Conditional generation can retain specified covariate effects, but depends on correctly specifying the conditional mechanism. Shuffled controls preserve observed block structure more directly, but can be invalid under sample dependence or unaccounted-for confounding.
| Approach | Useful when | Main risk |
|---|---|---|
| Independent generation | The target null is naturally represented by independent latent or observed components. | Unintentionally removes realistic within-block or cohort structure. |
| Conditional generation | A target relationship must be removed while covariate effects are retained. | Misspecified conditional mechanisms can create the wrong null. |
| Permutation control | Rows are exchangeable, and preserving observed values and within-block structure is important. | Unrestricted shuffling can break strata or dependence and may leave confounding. |
Diagnostics, Failure Modes, and Interpretation
Check the data-generating process and the resulting data. A generator audit asks whether the target coefficient or dependence was actually set to its null value and whether any shared latent variable, batch effect, sample identifier, or preprocessing step can carry the target signal through by another route. A data audit compares marginal distributions, missingness patterns, feature covariance within blocks, and covariate balance between the intended scenario and its control. These checks should be made before interpreting a method’s performance.
Also audit the analysis pipeline for leakage. If feature selection, tuning, or preprocessing uses outcome labels, the nominal outcome-null scenario can still produce optimistic findings. Apply the same pipeline decisions to control and non-control scenarios, and make clear which steps are repeated using only the training data in each replicate. A control that bypasses those steps may test a different procedure.
Under a complete global null, every reported target discovery is false. For replicate \(r\), let \(V_r\) be the number of false discoveries and \(R_r\) the number of reported discoveries. The false discovery proportion is \(V_r/\max(R_r,1)\); averaging this quantity across replicates estimates the false discovery rate for the specified simulation design. The proportion of replicates with \(R_r>0\) instead measures the chance of at least one discovery under that null. These summaries answer different questions. Under a partial null, some discoveries can be true, so false discoveries must be identified relative to the scenario’s known truth.
A particularly common mistake is to call any shuffled data “a null.” The reasoning is that changing sample order must remove all association. That is not generally correct: unrestricted permutation can retain chance association, fail to remove a relationship induced by strata, or create implausible pairs in clustered data. The correction is to state the exchangeability assumption, choose the permutation scheme to match it, and verify that the target relationship is disrupted while intended structure is preserved.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A no-shared-signal generator gives each omics block its own latent factor but includes a common batch effect. Does it guarantee no marginal cross-block association?
- A method makes at least one discovery in 12 of 200 complete-global-null replicates. What is the empirical proportion of replicates with at least one discovery?
- Why might shuffling outcomes across all samples be unsuitable when outcome distributions differ by clinic?
- What does a whole-block row permutation preserve, and what target structure is it intended to disrupt?
Solutions. (1) No. The common batch effect can induce marginal cross-block association even though the block-specific latent factors are independent. The scenario must distinguish absence of shared biological signal from absence of all observed association. (2) \(12/200=0.06\), or 6%. Under the complete null, each discovery is false, and this statistic estimates the probability of at least one discovery in this simulated setting. (3) A global shuffle changes clinic composition and may break the conditional outcome structure; use a justified within-clinic permutation or a conditional generator if exchangeability holds within clinics. (4) Applying the same row permutation to an entire block preserves its observed values, marginal distributions, and within-block covariance, while disrupting its original sample correspondence with the other block. It does not guarantee zero cross-block association in a single permutation.