Why Parameter Ranges Need Evidence
A simulation result applies to the data-generating conditions that were simulated. If a method performs well only under one convenient setting, that result does not establish that it will perform well in the range of studies where the method might be used. Parameter ranges make this scope visible: they describe which data conditions the benchmark treats as plausible, and which conditions it deliberately uses to probe failure.
Earlier material on scenario matrices and semi-synthetic benchmarks provides ways to organize scenarios and build data. Here the focus is choosing and justifying the values that those scenarios use. Relevant parameters can include cross-omics correlation, signal prevalence, effect size, noise level, missingness, and the relative scales of measurements. They are not interchangeable: each needs evidence appropriate to its scientific meaning and measurement scale.
A range is not a claim that every value is equally common, nor does it guarantee that every combination of parameter values is realistic. In a multi-omics simulation, a high signal prevalence, strong dependence, and low measurement noise might each be plausible separately while their simultaneous combination is rare or unsupported. Range selection therefore requires checking both individual parameters and combinations.
Build Ranges From Comparable Evidence
Start by defining the target application and the data quantity represented by each parameter. For example, a correlation between two assay features is not the same quantity as correlation between two latent factors; a reported missingness percentage may refer to samples, features, or individual measurements. Record the exact definition, units, scale, population, assay, and analysis behind every extracted value.
Prefer evidence that resembles the intended benchmark: similar tissue or population, assay platform, preprocessing, cohort design, and feature-selection process. A pilot dataset can be useful, but it may be small, selected, or affected by batch structure. Published estimates can broaden the evidence base, but differences in definitions or processing can make apparent agreement misleading. If studies are not sufficiently comparable, retain their context and consider separate scenario strata rather than pooling values mechanically.
For a continuous parameter, a practical first summary is the collection of comparable study estimates, not the minimum and maximum values reported anywhere. Extreme values may reflect sampling error, unusual populations, or incompatible measurement definitions. Choose a central setting using a justified summary, such as a median across comparable estimates, and choose bounds that cover the relevant evidence while representing uncertainty. For proportions, retain the denominator and distinguish a proportion of features from a proportion of samples. For parameters constrained by definition, such as a correlation, ensure the chosen bounds respect the valid scale.
Separate between-study variability from uncertainty in a study estimate. A confidence interval describes sampling uncertainty under a model for that study; it does not by itself describe the full range of conditions across target applications. Conversely, a range of study estimates is not automatically a confidence interval. State which source of variation the simulation is intended to represent.
Worked Example: Choosing a Cross-Omics Correlation Range
Suppose a hypothetical benchmark studies correlation between matched RNA and protein measurements for selected feature pairs. Three comparable pilot or published analyses report correlations of \(0.18\), \(0.31\), and \(0.27\). Assume they use the same correlation definition and broadly similar matched samples and preprocessing; for this illustration, treat the reported values as descriptive estimates rather than exact population values.
The central setting is the median of the three values. Ordering them gives \(0.18, 0.27, 0.31\), so the median is \(0.27\). The observed span is \(0.31-0.18=0.13\), but the span alone is not a principled range: three estimates may not cover the target applications, and their sampling uncertainty is not represented by the span.
For a transparent illustrative design, choose a plausible range of \(0.15\) to \(0.35\), broad enough to include all three estimates with modest allowance beyond them. The range is a design judgment, not a confidence interval. Include \(0.27\) as a central scenario, and use the lower and upper bounds to examine weaker and stronger dependence. If a separate \(0.05\) setting is added, label it a weak-dependence stress test unless evidence supports treating it as typical.
The interpretation depends on the stated feature-pair population. These summaries do not establish that every RNA-protein pair has correlations in this range, nor do they justify combining this correlation range with arbitrary signal prevalence and noise settings without checking whether those combinations are credible.
Translate Evidence Into a Scenario Set
A range becomes useful when translated into a manageable set of settings. For continuous parameters, a low, central, and high value can reveal broad patterns, but three points do not establish behavior at every point between them. For discrete or bounded parameters, select values that correspond to interpretable conditions, such as a low, moderate, and high proportion of active features. Include exact boundary values only when those boundaries represent a meaningful condition, not merely because they are mathematically possible.
When several parameters vary, avoid an unstructured Cartesian product if it generates implausible combinations or an unwieldy number of scenarios. One approach is to define a primary set of realistic joint scenarios based on evidence about dependencies, then add targeted sensitivity scenarios. For instance, if high missingness is associated with a particular cohort structure, represent that pairing rather than treating missingness as independent of cohort in every scenario. If joint evidence is absent, say so and treat combinations as hypothetical.
Worked Example: Setting a Signal-Prevalence Range
Suppose a hypothetical pilot has \(p=2{,}000\) analyzed features, of which \(k=80\) are classified as associated under a prespecified screening rule. The observed proportion is
This observed \(4\%\) is not automatically the true signal prevalence. The screening rule can miss real signals and include false discoveries, while feature selection may make the analyzed set unrepresentative of the original assay. Suppose, for illustration, that two additional comparable analyses give prevalence estimates of \(0.02\) and \(0.06\), using sufficiently similar definitions. The three descriptive values are \(0.02, 0.04, 0.06\); their median is \(0.04\).
A benchmark could use \(0.02\), \(0.04\), and \(0.06\) as low, central, and high settings, corresponding to \(40\), \(80\), and \(120\) active features out of \(2{,}000\), respectively. The counts follow directly from \(2{,}000q\). The range is defensible only if the three studies' signal definitions and feature universes are comparable; otherwise, the numbers should not be presented as a pooled empirical range. The benchmark report should name the screening rule and explain that this is an operational prevalence, not necessarily the fraction of all biologically nonzero effects.
Sensitivity Analysis: Does the Conclusion Depend on the Choice?
Sensitivity analysis varies uncertain or consequential design choices and checks whether the substantive conclusion changes. A useful primary question is not simply whether a performance statistic changes, but whether the method comparison changes: Does the ranking reverse? Does an apparent advantage become negligible? Does a method fail in a scientifically relevant region of the plausible range?
A one-at-a-time analysis varies one parameter while holding others at the central setting. It is easy to interpret and can identify influential parameters, but it can miss interactions. A small factorial set or carefully chosen joint scenarios can reveal whether the effect of one parameter depends on another. The latter is especially important when dependence, sparsity, noise, or missingness are related in the target data. Sensitivity settings should be selected before interpreting results; adding only settings that favor a preferred method invites biased conclusions.
Worked Example: Testing a Method Comparison Across a Range
Consider a hypothetical simulation that compares two integration methods using mean squared prediction error (MSE), measured in squared outcome units; lower is better. At the central noise setting, method A has MSE \(0.42\) and method B has MSE \(0.48\). Define the difference as \(\Delta=\text{MSE}_A-\text{MSE}_B\), so negative values favor A. Then
At a higher but still plausible noise setting, suppose the corresponding MSE values are \(0.71\) and \(0.73\), giving
At a prespecified high-noise stress test, suppose the values are \(0.94\) and \(0.91\), giving \(\Delta_{\text{stress}}=0.94-0.91=0.03\). Thus A has lower MSE at the central and plausible high-noise settings, but B has lower MSE in the stress test. These values do not establish a general winner; they show that the comparison is sensitive to noise severity and that the stress-test reversal should be reported as such. The conclusion should also account for simulation uncertainty, which is a separate issue from the range-selection question.
Justifying the Sensitivity Check
A simple way to quantify the importance of a range is to compare the method difference at its endpoints with the difference at the central setting. For a scalar performance contrast \(\Delta(\theta)\), define the change from the center as \(\Delta(\theta)-\Delta(\theta_c)\). If this change is large enough to alter the practical interpretation or sign of the comparison, the conclusion depends materially on \(\theta\). This is a descriptive diagnostic; it is not a hypothesis test and does not account for uncertainty unless uncertainty is separately estimated.
The reasoning is direct. A comparison claims some direction or magnitude for \(\Delta\) under a set of settings. If \(\Delta\) changes sign within that set, then the direction is not uniform over the set. If it retains its sign but changes substantially in magnitude, the directional conclusion may be more robust than the claim about practical importance. Reporting the values across the prespecified settings makes that distinction visible.
Diagnostics, Alternatives, and Limitations
Warning signs include ranges chosen without a target population, incompatible estimates combined without explanation, unexplained conversion between scales, and boundaries copied from the smallest and largest values ever reported. Another warning sign is a scenario grid in which every parameter is varied independently despite known relationships between them. After generating scenarios, check that each setting is valid, interpretable, and supported by the stated rationale. Compare simulated marginal summaries and dependencies with the evidence used to select the ranges; if they do not align, revise the assumptions or explain the mismatch.
There is no single best alternative to a range-based design. A narrow evidence-based range focuses on a well-defined application but may understate transportability concerns. A broad stress-test design probes failure more aggressively but can spend substantial effort on conditions that are not representative. Sampling parameters from an empirical distribution can represent variation within the observed evidence, but the result then depends on how that distribution was estimated and may underrepresent poorly observed tails. A hand-picked grid is transparent and easy to interpret, but can miss important interactions. These approaches answer different questions and can be combined when their roles are stated clearly.
Ranges inherit limitations from their sources. Small pilot datasets yield noisy summaries; selective publication may distort the evidence; preprocessing choices can shift estimates; and a parameter may vary across populations or platforms. Empirical plausibility also does not guarantee that a chosen data-generating model is realistic. Range justification is therefore one component of benchmark validity, not a substitute for checking the simulated data or for reporting uncertainty in method-performance comparisons.
Common Pitfall: Treating Observed Extremes as Natural Bounds
Incorrect reasoning: “The four studies reported values from \(0.12\) to \(0.39\), so the plausible range is exactly \(0.12\) to \(0.39\).” This assumes that the smallest and largest observed estimates are true limits. They may instead reflect sampling noise, differing cohorts, or inconsistent measurement definitions; the range may also omit values not observed in a small collection of studies.
Correction: Check comparability, sample size, and estimation uncertainty; explain how the target application relates to the sources; and choose bounds to represent the intended evidence and uncertainty. If the observed extremes are used as scenario values, describe them as observed estimates, not as natural limits. Add a clearly labeled stress test when useful, without implying it is empirically typical.
A Practical Documentation Sequence
Specify the intended application and exactly what the parameter measures, including its scale and denominator where relevant.
Record each source's population, assay, processing, estimate, uncertainty, and limitations; separate incompatible evidence rather than averaging it.
Justify a central setting and bounds, identify plausible and stress-test values, and document known parameter dependencies.
Evaluate prespecified settings and interactions, then report where the substantive comparison is stable and where it changes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Three comparable studies report a parameter estimate of \(0.10\), \(0.14\), and \(0.20\). Why is their observed minimum and maximum not automatically a definitive plausible range?
- A pilot identifies \(30\) active features among \(1{,}500\) analyzed features. Calculate the observed active-feature proportion.
- Suppose a method difference is positive at the central setting and negative at a plausible boundary. What does this imply about the comparison?
- Why might varying two parameters jointly be more informative than changing each one separately?
Solutions. (1) The estimates may be noisy or differ in population, measurement, or processing; the observed extremes do not establish natural bounds. Comparability and uncertainty must be assessed, and the range needs an explicit target-based rationale. (2) \(30/1{,}500=0.02\), or \(2\%\). This is an observed operational proportion and depends on how active features were defined. (3) The direction of the comparison changes across the examined settings, so the conclusion is not directionally robust over that range. (4) Parameters can interact or be associated in real data; a joint change can reveal behavior that one-at-a-time analyses, which hold other parameters fixed, may miss.