Why Examine Sensitivity to Simulation Assumptions?
A simulation comparison answers a conditional question: which integration strategy performs better under the data-generating assumptions that were specified? In multi-omics work, those assumptions may include the correlation between blocks, the number of outcome-associated features, and how signal is distributed across blocks. If a strategy ranks first under one plausible setting but not another, a single simulation scenario does not support a general claim that it is best.
Sensitivity analysis addresses this issue by deliberately varying uncertain assumptions and examining whether the relevant comparison changes. It does not establish that any simulated setting is the true one. Instead, it shows how conclusions depend on assumptions, and helps identify the conditions under which a strategy performs well or poorly.
The score must have a consistent direction. For an accuracy metric where larger values are better, use the metric directly. For a loss or computational cost where smaller values are better, reverse the sign or define the contrast so that its direction is explicit. Cost and predictive accuracy may be different outcomes and should not be combined into an arbitrary single score unless the combined utility has been justified.
Designing Meaningful Sensitivity Scenarios
Begin by identifying which assumptions are uncertain and why. Then specify a plausible range or a small set of scientifically interpretable settings before looking at the strategy ranking. For example, low, moderate, and high cross-block correlation can represent different degrees of redundancy between assays. Sparse and less sparse outcome signals represent different numbers of associated features. The settings should be plausible for the intended application, not chosen only because they make one strategy look favorable.
A useful scenario changes the intended assumption while keeping other features controlled when the goal is to isolate that assumption. For instance, when varying cross-block correlation, preserve the marginal distributions, outcome signal strength, sample size, and feature counts if the simulation design permits it. This requires care: changing a covariance matrix can also change the signal-to-noise ratio or the marginal association with the outcome. Check the generated data rather than assuming the nominal parameter had only the intended effect.
If two assumptions may interact, vary them jointly. A one-at-a-time analysis can reveal how changing correlation affects results at a fixed sparsity level, but it cannot show whether that effect differs for sparse and dense signals. A small factorial grid is often an interpretable way to expose such dependence without making the scenario set unwieldy.
Worked Example: Varying Cross-Block Correlation
Suppose a hypothetical simulation compares early integration, which combines the blocks before fitting, with late integration, which combines block-specific predictions. The outcome is a higher-is-better prediction score. The simulated design is intended to keep sample size, feature counts, marginal outcome associations, and signal sparsity fixed while varying the cross-block correlation \(\rho\). Across 200 independent replicates per setting, suppose the mean scores are as follows.
| Cross-block correlation \(\rho\) | Early integration | Late integration | Early minus late |
|---|---|---|---|
| 0.10 | 0.76 | 0.72 | 0.04 |
| 0.50 | 0.78 | 0.75 | 0.03 |
| 0.80 | 0.77 | 0.78 | -0.01 |
At \(\rho=0.10\), the estimated contrast is \(0.76-0.72=0.04\), favoring early integration by 0.04 score units. At \(\rho=0.80\), it is \(0.77-0.78=-0.01\), favoring late integration by 0.01 score units. Thus, the ranking is not stable over these three settings: the mean-score comparison reverses at high correlation. It would be inaccurate to report only the first row and conclude that early integration is generally superior.
The interpretation is conditional: in this hypothetical design, the mean-score ranking changes as the specified cross-block correlation changes, with the other intended features held fixed. Before attributing the change to \(\rho\), inspect the generated marginal associations and signal-to-noise ratios to confirm that they remained comparable as intended. The scores alone do not prove that correlation is the sole cause if other properties also shifted.
Replication, Pairing, and Precision
For each scenario, simulation replicates estimate expected performance. When comparing two strategies, one can apply both strategies to the same generated data in each replicate and calculate a replicate-level difference. This matched design controls some replicate-to-replicate variation, but it does not automatically make the comparison more precise.
Let \(A_r\) and \(B_r\) be the scores for strategies \(A\) and \(B\) on replicate \(r\), and let \(D_r=A_r-B_r\). If the replicate-level scores have variances \(\sigma_A^2\) and \(\sigma_B^2\), and covariance \(\sigma_{AB}\), then
The unpaired difference between independent sets of scores has variance \(\sigma_A^2+\sigma_B^2\), at the single-replicate scale. Therefore, pairing reduces variance when \(\sigma_{AB}>0\), leaves it unchanged when \(\sigma_{AB}=0\), and increases it when \(\sigma_{AB}<0\). The precision gain depends on the covariance of the strategies' scores; shared data alone does not guarantee it.
Worked Example: Checking the Effect of Pairing
Suppose a hypothetical comparison uses \(R=100\) matched simulation replicates. Each strategy's replicate scores have standard deviation \(0.06\), so \(\sigma_A^2=\sigma_B^2=0.0036\). Suppose their correlation across matched replicates is \(0.50\). Their covariance is \(0.50(0.06)(0.06)=0.0018\). The standard error of the mean paired difference is therefore
For independent, unpaired scores with the same marginal variances, the standard error would be
Here the positive covariance makes pairing more precise. If instead the correlation were \(-0.25\), then \(\sigma_{AB}=-0.25(0.06)(0.06)=-0.0009\), giving
In that case, the paired standard error exceeds the unpaired value of about \(0.00849\). Pairing is a design choice whose precision should be checked from the observed replicate-level covariance, not assumed.
When Assumptions Interact
The effect of changing one assumption can depend on another. To see this, compare strategy contrasts across a two-by-two grid of cross-block correlation and signal sparsity. A contrast that changes little across correlation at one sparsity level could change substantially at another. This is a strategy-by-scenario interaction: the relative performance of the strategies depends jointly on the scenario settings.
Worked Example: Correlation and Sparsity Together
In a hypothetical benchmark, let a sparse signal mean that 5% of features carry outcome signal and a denser signal mean that 40% do. Let low and high correlation be \(\rho=0.10\) and \(\rho=0.70\). The table shows the mean-score contrast \(\Delta_{A,B}\), in score units, for strategies \(A\) and \(B\).
| Signal sparsity | \(\rho=0.10\) | \(\rho=0.70\) | Change in contrast as \(\rho\) increases |
|---|---|---|---|
| 5% signal features | 0.04 | 0.03 | \(0.03-0.04=-0.01\) |
| 40% signal features | 0.02 | -0.03 | \(-0.03-0.02=-0.05\) |
The difference-in-differences is \((-0.05)-(-0.01)=-0.04\) score units. Thus, increasing correlation changes the strategy contrast 0.04 units more in the denser-signal setting than in the sparse-signal setting. The contrast remains positive in both sparse settings, but changes sign in the high-correlation, denser-signal setting. Reporting only the average contrast across all four cells could conceal this conditional reversal.
This calculation describes a pattern in the simulated mean scores; it does not by itself identify a biological mechanism. As in the previous example, attribution depends on whether the simulation actually held other intended properties fixed. If conclusions are important, report the cell-specific contrasts and their Monte Carlo uncertainty rather than only a pooled summary.
Assumptions, Diagnostics, and Alternatives
Sensitivity conclusions are conditional on the simulation design and the range of scenarios examined. A useful analysis makes those conditions explicit:
- The data-generating mechanism is implemented as specified, and its realized properties are checked.
- The chosen scenario values represent plausible or deliberately informative alternatives for the application.
- Within each scenario, the number of independent replicates is sufficient to estimate the score contrasts with useful precision.
- The metric and its direction are defined consistently across strategies and scenarios.
Diagnostics should include the realized cross-block correlations, the realized fraction of signal features, and the marginal outcome associations or signal-to-noise ratios that were intended to remain fixed. Also inspect replicate-level score distributions and uncertainty for the contrasts. If a nominal correlation change also alters the marginal feature distributions, or if the score estimates are noisy enough that a small apparent reversal is uncertain, describe that limitation rather than presenting the ranking as definitive. Monte Carlo uncertainty quantifies simulation error for the specified scenarios; it does not resolve uncertainty about whether those scenarios are realistic.
One-at-a-time variation is easy to interpret and useful for screening, but can miss interactions. A factorial grid estimates joint patterns and interactions, but requires more simulated settings and can become expensive. Probabilistic sensitivity analysis assigns distributions to uncertain assumptions and summarizes performance over those distributions; its results then depend on the chosen distributions. Stress testing examines deliberately challenging settings, but those settings should not be mistaken for typical conditions. These approaches answer different questions and can be combined when resources permit.
Sensitivity analysis is most useful when it leads to a qualified, reproducible conclusion: for example, “strategy A has higher mean score under sparse signals across the examined correlation range, while strategy B performs better for dense signals at high correlation.” It is less informative to claim that one method is universally best based on a small, selectively chosen set of settings. The earlier material on strategy-by-scenario interactions and Monte Carlo uncertainty provides useful concepts for describing these patterns without treating noisy rankings as certain.
Common Mistake
Check Your Understanding
Use these questions to check whether you can plan and interpret a sensitivity analysis. Brief solutions follow each question.
- What does a sign change in \(\Delta_{A,B}(\theta)\) across plausible scenarios tell you? Solution: The mean-score ranking reverses over the examined scenarios, so a claim that one strategy is uniformly better is not supported across that range.
- Why should marginal outcome associations be checked when varying cross-block correlation? Solution: The correlation manipulation may unintentionally alter signal strength or other data properties. If those change too, the observed performance change cannot be attributed to cross-block correlation alone.
- Two paired strategies have replicate-score covariance \(-0.001\). Does pairing necessarily reduce the variance of their score difference? Solution: No. Since the variance contains \(-2\sigma_{AB}\), a negative covariance increases the variance relative to an unpaired difference with the same marginal variances.
- In a two-factor sensitivity analysis, the effect of correlation on the strategy contrast is \(-0.01\) at sparse signal and \(-0.05\) at dense signal. What is the difference-in-differences? Solution: \(-0.05-(-0.01)=-0.04\) score units. The correlation effect is 0.04 units more negative at dense signal.
- Does Monte Carlo uncertainty show whether the simulated scenario is realistic? Solution: No. It describes uncertainty from finite simulation replicates under the specified data-generating assumptions; realism requires separate scientific justification.