Why Sample Size Needs Its Own Benchmark
An integration strategy may perform differently when it has data from 80 subjects than when it has data from 800. More subjects can make outcome-related patterns easier to estimate, but they do not guarantee better prediction: noisy measurements, high-dimensional blocks, overfitting, and model choices can all affect the result. A useful benchmark therefore asks a specific question: how does performance change as the number of subjects changes, when the other simulation conditions are held fixed?
The target is not a universal ranking of early, intermediate, and late integration. It is a comparison under a stated data-generating scenario and analysis plan. The earlier material on fair comparison targets and simulation design supplies the basis for making such comparisons. Here the focus is the subject count: how to vary it, what to measure across repeated runs, and how to interpret differences without confusing an observed trend with a certain or universal conclusion.
Constructing a Controlled Comparison
First state which subjects contribute to \(n\). In a prediction benchmark, one transparent design fixes an independent test set and varies only the size of the training cohort. Each simulated training cohort contains matched measurements for all omics blocks and the same outcome for each subject. Every strategy is trained on that cohort and evaluated on the same test subjects. The test set must not contribute to fitting, feature selection, preprocessing, or tuning. If performance is intended to generalize beyond one particular test set, the simulation should also vary test-set generation or use sufficiently many independent evaluation subjects; a fixed test set estimates performance conditional on that set.
The values of \(n\) should be chosen before inspecting results and should cover a meaningful range. For example, \(n=100, 200, 400,\) and \(800\) can show both smaller-cohort behavior and larger-cohort behavior. The precise values depend on the application and the computational budget. Keep the number and dimensions of blocks, the signal-generating process, noise distributions, and outcome form fixed across this comparison. Likewise, apply the same preprocessing and tuning rules at each \(n\), unless the benchmark explicitly defines a different policy as part of the question.
Use repeated, independently generated training cohorts at each sample size. Within a replicate, compare the strategies on the same simulated subjects and evaluation data. This pairing prevents a strategy from appearing better merely because it happened to receive an easier simulated cohort. Replicates are repetitions of the benchmark, not additional subjects pooled into one fitted model.
State the prediction or other performance target, the sample sizes, and whether \(n\) means training subjects or all simulated subjects.
Set the omics blocks, outcome mechanism, noise, and analysis and evaluation policies. Change the training subject count, not these other factors.
Generate independent replicates; within each replicate fit each strategy on the same cohort and evaluate each on the same held-out subjects.
At each \(n\), report average performance and its replicate-to-replicate spread or uncertainty. Compare strategies within replicates as well as across sample sizes.
What the Replicates Estimate
Let \(L_{jrn}\) denote a loss for strategy \(j\), replicate \(r\), and training size \(n\), where smaller values are better. For \(R\) independent replicates at a given \(n\), the estimated average loss is \(\bar L_{jn}=\sum_{r=1}^{R}L_{jrn}/R\). The sample standard deviation across those replicate losses describes how much the observed loss varies across simulated cohorts and evaluation data under the design. It is not the standard error of the estimated average.
If replicate losses are independent and have finite variance, the standard error of their average is the sample standard deviation divided by \(\sqrt{R}\). This measures Monte Carlo precision for the average under the simulation design. It does not measure the variability of a future fitted model’s loss, which is represented more directly by the replicate spread. In practice, an estimate based on very few replicates can be unstable, so both the number of replicates and the uncertainty of summaries matter.
For two strategies, pairing gives a direct comparison. Define \(d_r=L_{Arn}-L_{Brn}\). Positive \(d_r\) favors strategy B for a loss metric. The average difference is \(\bar d\), and the standard error of that average is \(s_d/\sqrt{R}\), where \(s_d\) is the sample standard deviation of the replicate differences. Because the strategies face the same replicate conditions, this comparison focuses on their difference rather than on variation in the difficulty of cohorts.
Worked Example: Isolating the Sample-Size Change
Suppose a hypothetical continuous-outcome benchmark compares early integration at \(n=100\) and \(n=400\). The outcome-generating mechanism, block dimensions, noise, tuning policy, and independent test set are fixed. Each entry below is test-set mean squared error (MSE), in squared outcome units, from one of five paired replicates.
| Replicate | MSE at \(n=100\) | MSE at \(n=400\) | Difference: \(n=100\) minus \(n=400\) |
|---|---|---|---|
| 1 | 1.20 | 0.95 | 0.25 |
| 2 | 1.10 | 0.90 | 0.20 |
| 3 | 1.30 | 1.00 | 0.30 |
| 4 | 1.00 | 0.85 | 0.15 |
| 5 | 1.15 | 0.95 | 0.20 |
The average MSE at \(n=100\) is \((1.20+1.10+1.30+1.00+1.15)/5=5.75/5=1.15\). At \(n=400\), it is \((0.95+0.90+1.00+0.85+0.95)/5=4.65/5=0.930\). The paired mean reduction is \((0.25+0.20+0.30+0.15+0.20)/5=1.10/5=0.220\) squared outcome units. Relative to 1.15, that reduction is \(0.220/1.15\approx0.191\), or about 19.1%.
In this hypothetical benchmark, the larger training size has lower average test MSE. This is a descriptive result for the stated scenario, not proof that increasing \(n\) always improves performance. Five replicates provide limited information about the stability of the estimated reduction; the paired results should not be presented as a definitive general ranking.
Comparing Strategies at a Given Sample Size
The comparison between strategies should be made within each sample size, because a strategy’s relative advantage can change as data become more plentiful. For two strategies evaluated on the same replicates, calculate the replicate differences first, then summarize their average and spread. With a small number of replicates, these summaries are especially uncertain. A confidence interval for the mean paired difference can be formed using a \(t\) procedure when the replicate differences are independent and their distribution is reasonably compatible with that procedure; its assumptions should not be treated as automatic.
Worked Example: A Paired Difference With Correct Arithmetic
At a fixed sample size, suppose a hypothetical benchmark records the losses below for strategies A and B. Let \(d_r=L_{Ar}-L_{Br}\), so a positive difference favors B because lower loss is better.
The paired differences are \(d=(0.10,0.05,-0.05,0.10)\). Their mean is \(\bar d=(0.10+0.05-0.05+0.10)/4=0.20/4=0.05\) loss units. Deviations from the mean are \(0.05,0,-0.10,0.05\), so the sum of squared deviations is \(0.0025+0+0.0100+0.0025=0.0150\).
The sample standard deviation and standard error are therefore
For illustration, a two-sided 95% \(t\) interval with \(df=3\) uses the critical value approximately 3.182. Its limits are \(0.05\pm3.182(0.03536)\), or approximately \([-0.0625,0.1625]\) loss units. The interval includes zero, so these four replicates do not establish a clear direction of the mean difference at this level. They also do not demonstrate equivalence: the interval permits differences in either direction. The calculation assumes independent replicate differences and a \(t\)-compatible distribution; with only four replicates, that distributional assumption is difficult to assess.
Performance Spread Is Not Precision of the Average
Sample size can affect both the center and the spread of performance. A larger training cohort may reduce variability across fitted models, but the benchmark must measure that rather than assume it. Meanwhile, the precision of an estimated average depends on the number of simulation replicates \(R\), as well as the spread across replicates. Increasing the number of subjects per replicate and increasing the number of replicates answer different questions.
Worked Example: Separating Spread From Standard Error
At \(n=200\), suppose four hypothetical replicate MSE values for one strategy are \(1.0, 1.1, 0.9,\) and \(1.0\). Their mean is \(4.0/4=1.00\). The squared deviations from 1.00 sum to \(0+0.01+0.01+0=0.020\), giving a sample standard deviation of \(\sqrt{0.020/3}\approx0.08165\) MSE units and a standard error of \(0.08165/\sqrt{4}\approx0.04082\) MSE units.
At \(n=800\), suppose the four losses are \(0.8,0.9,0.7,\) and \(0.8\). Their mean is \(0.80\); the squared deviations again sum to \(0.020\), so the sample standard deviation is again about \(0.08165\) and the standard error about \(0.04082\). These hypothetical results show a lower average loss at \(n=800\), but no reduction in observed replicate spread. They do not show that the fitted models are less variable at the larger sample size. Also, because \(R=4\), the estimated standard errors themselves are imprecise.
Diagnostics, Alternatives, and Limitations
Plot performance against \(n\), showing replicate-level values or intervals as well as the average. Look for unusually influential replicates, abrupt changes, and patterns suggesting that an implementation or analysis rule changed across sample sizes. Check that each replicate contains the intended number of matched subjects and that preprocessing, feature selection, and tuning use training data only. If one strategy fails to fit at some sample sizes, record the failure rate rather than silently removing those replicates; excluding failures can make a method look artificially successful.
There are several design alternatives, each answering a different question. Independent cohorts at every \(n\) make each sample size a separate draw from the data-generating process. Nested cohorts, where smaller training sets are subsets of larger ones, create a direct within-replicate sample-size trajectory, but performance values across \(n\) are then dependent by construction. Either can be appropriate if the dependence is acknowledged and comparisons are made accordingly. Unpaired strategy runs are possible, but they usually waste information when strategies could have been evaluated on the same replicate conditions.
A single fixed train-test split is computationally simple but can make conclusions overly dependent on that split. Repeated independent evaluation sets can better represent evaluation-sample variation, at additional computational cost. In either design, test observations must remain separate from every model-development step. The estimated uncertainty is conditional on the specified simulation mechanism and analysis policy; it does not account for mechanisms omitted from the benchmark.
A sample-size curve is also scenario-specific. Holding other factors fixed isolates the effect of changing \(n\) in that scenario, but it does not establish what happens under different signal strengths, block dimensions, or noise structures. Those factors require separate controlled comparisons. Nor does a downward trend prove that a strategy has reached its best possible performance: a plateau may reflect the particular model, signal, metric, or sample sizes examined.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A benchmark varies the training cohort size but also doubles the number of omics features at the largest size. What can no longer be attributed solely to sample size?
- For a lower-is-better loss, define a paired difference that makes positive values favor strategy B.
- Four independent replicate differences have \(s_d=0.06\) loss units. What is the standard error of their mean?
- Explain why a small standard error across simulation replicates does not imply that an individual fitted model will have nearly the mean loss.
Solutions. (1) The change in performance is confounded: both subject count and feature dimension changed, so their separate contributions cannot be identified from that comparison. (2) Use \(d_r=L_{Ar}-L_{Br}\); positive values mean A has greater loss and therefore favor B. (3) \(SE(\bar d)=s_d/\sqrt{R}=0.06/\sqrt{4}=0.030\) loss units. (4) The standard error describes precision of the estimated mean across replicates. Variation in an individual model’s loss is described by the replicate-to-replicate spread, which can be much larger than the standard error.