Worked Benchmark With Unequal and Redundant Blocks

Why Unequal and Redundant Blocks Need a Deliberate Benchmark

A multi-omics benchmark can produce a convincing ranking for the wrong reason. A block with many features may appear influential because it contains more columns, even when those columns repeat nearly the same information. Conversely, a smaller block may carry a strong but narrow signal. To interpret a comparison of early, intermediate, and late integration, the benchmark should make block size, redundancy, and signal strength explicit and vary them in controlled ways.

The goal here is not to declare one strategy universally best. It is to determine what a comparison says under a specified data-generating process, and whether its main conclusion survives targeted changes to that process. The generator, the evaluation metric, and the comparison across repeated datasets must all be interpretable.

A Transparent Generator for Unequal Blocks

Consider a continuous outcome and three complete omics blocks measured on the same individuals. Let \(U_A,U_B,U_C\) be independent latent block signals, each with mean 0 and variance 1. Within block \(b\), feature \(j\) is generated as \(X_{bj}=\sqrt{\rho}\,U_b+\sqrt{1-\rho}\,\epsilon_{bj}\), where the \(\epsilon_{bj}\) are independent \(N(0,1)\) errors, independent of all latent signals. Thus each feature has variance 1, and any two different features in that block have correlation \(\rho\). Set \(\rho=0.75\), and let the block sizes be \(p_A=4\), \(p_B=12\), and \(p_C=2\).

Generate the outcome as \(Y=0.90U_A+0.55U_B+0.25U_C+\epsilon_Y\), where \(\epsilon_Y\sim N(0,1)\) is independent of the features and latent signals. The coefficients describe different signal strengths: block A has the strongest latent association with the outcome, block B a moderate one, and block C the weakest. They are not direct regression coefficients for each observed feature, because the features are noisy measurements of the latent signals.

This construction has clear assumptions: individuals are independent; the latent signals are independent; the within-block feature errors are independent; and the outcome noise is independent, homoscedastic, and Gaussian. For a prediction benchmark, use independent training and test individuals generated from the same specified process. Any scaling or fitting is performed using training data, and each strategy receives the same training data and evaluation target. These assumptions make the benchmark easy to inspect, but they are not claims about real omics data.

Compare Strategies on a Common Basis

In a useful benchmark, the integration strategies address the same prediction task and are judged by the same metric. For example, an early strategy can concatenate the standardized blocks and fit one predictor; an intermediate strategy can fit block-specific predictors and combine their outputs in a second model; a late strategy can combine block-specific predictions by a prespecified rule. The precise algorithms may vary, but model flexibility and tuning budgets should be matched as closely as the design allows, consistent with the earlier material on integration benchmarks.

Here use test-set root mean squared error (RMSE), in outcome units. For \(n_{\text{test}}\) test observations with outcomes \(y_i\) and predictions \(\hat y_i\),

Smaller RMSE indicates smaller average squared prediction error on the evaluated test individuals. RMSE does not by itself reveal whether predictions are calibrated, whether errors differ by subgroup, or whether a model will transfer to a different population. Those are separate questions; do not treat a single metric as a complete description of predictive performance.

Use Controlled Variants to Locate the Source of a Ranking

A baseline comparison is informative only relative to its design. If one strategy performs better, the result could depend on the unequal number of columns, the redundant structure, or the uneven block-level signal. Construct variants that change one feature of the generator at a time. For example, retain only four of block B's 12 features to equalize block sizes; reduce within-block correlation to examine less redundant features; or set the three signal coefficients equal to examine a more balanced signal. Keep the outcome definition and test metric unchanged.

In practice, retaining four features from a redundant block can change information slightly, as the variance calculation above shows. Likewise, reducing correlation changes how much information is available per block. A controlled variant is therefore a targeted sensitivity analysis, not a magical intervention that leaves every consequence unchanged. State what is changed and what may change as a result.

Justification for the Paired Comparison

Let \(D_r\) be the difference in test RMSE between two strategies in simulation replicate \(r\), with the same generated dataset used for both strategies. The estimated mean difference is \(\bar D=\frac{1}{R}\sum_{r=1}^R D_r\), where \(R\) is the number of replicates. If the replicate differences are independent draws with finite variance \(\sigma_D^2\), then the variance of their mean is \(\sigma_D^2/R\), because variances of independent terms add and each term is divided by \(R\). Replacing \(\sigma_D\) by the sample standard deviation \(s_D\) gives \(SE(\bar D)=s_D/\sqrt{R}\).

When \(R\) is sufficiently large and no few replicates dominate, a normal approximation gives \(\bar D\pm1.96SE(\bar D)\) as an approximate 95% interval for the mean difference under the generator. The approximation concerns Monte Carlo replicates; its reliability depends on the distribution of \(D_r\) and the number of replicates. The paired design usually improves precision because shared dataset difficulty affects both strategies and is partly removed in \(D_r\).

Diagnostics, Alternatives, and Limitations

Check the benchmark itself before interpreting a ranking. Verify that simulated feature variances and within-block correlations are close to their targets, and that the generated outcome has the intended block-level associations. Confirm that all strategies use the same replicate datasets, training and test partitions, outcome definition, and evaluation metric. Inspect replicate-level differences as well as their means: a small mean can conceal large variability or a few influential replicates.

A useful diagnostic is to examine block contributions or ablations, removing one block at a time under the same evaluation design. If removing a nominally strong block scarcely changes performance, that may indicate that another block carries redundant signal, or that the fitted strategy does not exploit the block as intended. A result that changes sharply when the feature count is equalized may reflect sensitivity to block dimension, but it does not prove that the larger block was merely “overweighted”; the retained features and model regularization also matter.

Alternatives to the single generator include a factorial set of scenarios that varies size, redundancy, and signal strength systematically, or a design that deliberately varies one factor while fixing the others. The first can reveal interactions but requires more simulations and careful interpretation; the second is easier to attribute but may miss interactions. One can also report several metrics, such as calibration alongside discrimination when appropriate, but metrics answer different questions and should not be collapsed into one ranking without justification.

A synthetic benchmark is limited by its assumptions. Real omics blocks may have complex dependence within and between blocks, missing measurements, batch effects, nonlinear signals, and sample sizes that are small relative to feature counts. A favorable result under this generator is evidence about this controlled setting, not evidence of causal effects or guaranteed deployment performance. The appropriate conclusion is conditional: identify which strategy performs well, on which outcome metric, under which block structure, and whether the comparison is stable across the controlled variants.

Common Mistakes

• Equating more columns with more information. Twelve highly correlated features need not carry three times the distinct information of four features; quantify or vary redundancy rather than infer information from dimension.
• Changing several generator properties at once. If block size, correlation, and signal strength all change together, a changed ranking cannot be attributed to any one of them.
• Comparing unpaired uncertainty. Separate strategy-level standard deviations do not replace the variability of within-replicate differences when the same datasets are used.
• Calling a small numerical difference a winner. Check its uncertainty and whether it persists across scenarios; a rank alone does not describe the size or stability of a difference.
• Generalizing beyond the generator. A simulation ranking is conditional on the specified data-generating process and evaluation design, not a universal ordering of integration strategies.