Why a Benchmark Needs a Scenario Matrix
A method comparison is informative only if its simulated conditions let us answer the benchmark question. A single simulated data set cannot show whether performance changes with sample size, feature dimensions, cross-omics dependence, signal structure, or data quality. A scenario matrix makes those conditions explicit: each row represents one combination of factor levels under which the methods will be compared.
The matrix is a design for a benchmark, not a recipe for generating the data. Its purpose here is to specify which conditions are compared and which are held fixed. The earlier material on Choosing Benchmark Estimands establishes what performance quantity is to be compared; the matrix makes clear over which conditions that comparison will be assessed.
In this tutorial, the principal factors are sample size, feature dimensions across modalities, cross-omics dependence, signal structure, and data quality. Some factors are numerical, such as \(n\) or the number of features in a modality. Others are categorical, such as “weak” versus “strong” dependence or “shared” versus “modality-specific” signal. Categories need not have a natural numerical order. Even for numerical factors, treating levels as categories is often the safest way to compare a small set of prespecified conditions without assuming a linear trend.
Designing Factors and Comparisons
Before listing scenarios, specify the benchmark question and identify the factors that could change the performance quantity of interest. For each factor, choose levels that are interpretable and relevant to the question. Also name important settings that are not being varied. These fixed settings should be held constant across scenarios unless the design explicitly treats them as factors.
For feature dimensions, record the dimension of each modality, not just an overall total, when the balance between modalities matters. For data quality, state what the labels mean in operational terms—for example, “complete” versus “moderately noisy”—rather than relying on vague labels such as “good” and “bad.” Detailed definitions of data-generating mechanisms belong to the simulation design; the matrix must at least make the intended distinction unambiguous.
A fully crossed matrix includes every permitted combination of factor levels. With \(k\) factors and \(L_j\) levels for factor \(j\), the full crossing has \(\prod_{j=1}^{k} L_j\) scenarios. Full crossing makes comparisons transparent and allows interactions—situations where the effect of one factor depends on another—to be examined. It can, however, become computationally expensive. A smaller design can be reasonable, but its omissions and limitations should be explicit.
Do not force combinations that are scientifically incoherent or impossible merely to achieve a rectangular table. Instead, define the set of permissible combinations and explain the restriction. But do not use “infeasible” as a convenient reason to omit combinations that are essential to separate the benchmark’s target factors. The design should distinguish a genuine constraint from an accidental coupling of factors.
Worked Designs
Worked Example: Counting a Fully Crossed Matrix
Suppose a hypothetical benchmark varies five binary factors: sample size (\(n=100\) or \(n=400\)); feature dimensions (lower or higher); cross-omics dependence (weaker or stronger); signal structure (shared or modality-specific); and data quality (complete or noisier). Assume every combination is meaningful and feasible. How many scenarios are required for a full crossing?
Each factor has two levels, so the number of distinct combinations is
Thus, the matrix has 32 scenario rows. If three methods are evaluated in every scenario using 100 Monte Carlo repetitions per method and scenario, the planned number of method-repetition fits is
This count assumes that every method is run in every repetition and that no fits are lost or excluded. The 9,600 fits are not 9,600 distinct scenarios: there are 32 conditions, each evaluated repeatedly. If the cost is too high, the design decision is which comparisons can be reduced without obscuring the benchmark question—not whether to silently leave some factor combinations unrepresented.
Worked Example: A Balanced Fraction With a Stated Limitation
A hypothetical team cannot afford all 32 combinations above and proposes eight scenarios. It wants each binary factor represented at both levels and every pair of factors to appear in all four level combinations equally often. Let \(-\) and \(+\) denote the two levels of each factor. The following eight rows meet that pairwise balance requirement:
| Scenario | Sample size | Dimensions | Dependence | Signal | Quality |
|---|---|---|---|---|---|
| 1 | − | − | − | − | − |
| 2 | − | − | + | + | + |
| 3 | − | + | − | + | − |
| 4 | − | + | + | − | + |
| 5 | + | − | − | + | + |
| 6 | + | − | + | − | − |
| 7 | + | + | − | − | + |
| 8 | + | + | + | + | − |
Each factor has four \(+\) and four \(-\) entries. To check pairwise balance, consider sample size and dimensions: the four combinations \((-, -),(-,+),(+,-),(+,+)\) occur in scenarios 1–2, 3–4, 5–6, and 7–8, respectively. The same check applies to every pair of columns; each combination occurs twice. This is a useful screening design when the goal is to avoid simple imbalance among pairs of factors.
It is not equivalent to the full crossing. Eight rows cannot represent all 32 combinations, so higher-order interactions may be inseparable from one another or from main effects in the chosen design. Pairwise balance alone does not establish that all interactions are estimable. The team should therefore restrict its conclusions to comparisons supported by the design and should not claim that it has assessed every joint condition.
Worked Example: Diagnosing Confounding in a Four-Scenario Matrix
Consider a hypothetical benchmark that varies sample size and feature dimension. Its original matrix has only two conditions: \(n=100\) with 100 features, and \(n=400\) with 1,000 features. Suppose the observed error rates are 0.24 and 0.11, respectively. The error rate is a proportion, so these values are dimensionless.
The observed difference is
This is a 0.13 lower error rate in the second condition, but it cannot be attributed specifically to larger sample size: feature dimension changed at the same time. Nor can it be attributed specifically to lower feature dimension, because that changed alongside sample size. The two factors are confounded in this comparison.
A repaired, fully crossed matrix adds the missing combinations. Suppose its hypothetical error rates are:
| Sample size | 100 features | 1,000 features |
|---|---|---|
| \(n=100\) | 0.24 | 0.17 |
| \(n=400\) | 0.18 | 0.11 |
At 100 features, changing sample size from 100 to 400 changes error by \(0.18-0.24=-0.06\). At 1,000 features, the same sample-size contrast is \(0.11-0.17=-0.06\). Conversely, the feature-dimension contrast is \(0.17-0.24=-0.07\) at \(n=100\), and \(0.11-0.18=-0.07\) at \(n=400\). These differences are now separately calculable at fixed levels of the other factor. The matching differences in this hypothetical table suggest no interaction on the error-rate difference scale; they do not prove that interactions are absent in other settings.
Why Crossing Helps: A Short Justification
For two factors with two levels each, let \(Y_{ij}\) denote the expected performance metric at factor levels \(i\) and \(j\), where each index is either 0 or 1. A simple representation is
Here \(\beta_A\) and \(\beta_B\) are main-effect terms under this coding, while \(\beta_{AB}\) represents an interaction. In a full crossing, the effect of changing factor A at level \(j\) of factor B is
This calculation shows why both levels of B matter: the effect of A can differ across them if \(\beta_{AB}\) is not zero. If the design observes only \((i,j)=(0,0)\) and \((1,1)\), its difference is \(Y_{11}-Y_{00}=\beta_A+\beta_B+\beta_{AB}\). The contrast combines the two main effects and their interaction; the separate contributions cannot be recovered from that difference alone. A complete crossing provides the cell comparisons needed to distinguish those patterns, subject to adequate replication and the chosen analysis.
Alternatives, Diagnostics, and Limits
A full factorial design is the clearest option when all combinations are feasible and computationally affordable. A one-factor-at-a-time design varies each factor around a baseline while fixing the others there. It can answer narrow baseline comparisons, but it omits combinations in which factors change together and cannot reveal whether a factor’s effect depends on another factor. Fractional factorial or other carefully balanced designs reduce the number of conditions; they are useful when cost is binding, but their ability to distinguish effects depends on the specific design and assumptions about interactions. No reduced design should be described as a full assessment of all combinations.
Before running a benchmark, inspect the matrix as a design rather than just a list. The following checks catch common problems:
- Coverage: verify that every target factor has its intended levels and that important combinations are present.
- Balance: count how often each level and, where relevant, each pair of levels occurs. Balance does not guarantee that every effect is estimable, but imbalance is a warning to investigate.
- Confounding: check whether one factor is always paired with a particular level of another. If so, their effects may not be separable.
- Feasibility: flag combinations that are impossible or scientifically incoherent, and document the restriction rather than silently dropping rows.
- Redundancy: remove duplicate rows unless repeated conditions are deliberately included for a documented reason.
- Comparability: make sure methods being contrasted are evaluated under the same scenario definitions and with the same stated performance metric.
A matrix is not a guarantee of validity. Chosen levels may not cover realistic conditions; qualitative categories may hide important variation; omitted factors can still influence results; and a large full crossing can be costly. Monte Carlo variability can also make observed scenario-specific performance uncertain. Report that uncertainty alongside performance rather than treating a single simulated estimate as exact. The matrix supports interpretation only within the conditions it actually covers.
Check Your Understanding
Use the factor definitions and comparison principles above. Solutions are included so you can check both the arithmetic and the design reasoning.
- A matrix crosses three levels of sample size, two feature-dimension settings, and two data-quality settings. How many scenarios are in the full crossing?
- A benchmark compares \(n=150\) with 200 features against \(n=600\) with 2,000 features. Which factor-specific conclusions are supported by that contrast?
- In a full crossing of sample size and data quality, error rates at the two quality levels are 0.20 and 0.12 for \(n=100\), and 0.15 and 0.10 for \(n=400\). Calculate the quality contrast at each sample size. Does the contrast match across sample sizes?
- What can pairwise balance in a fractional design establish, and what does it not establish by itself?
Solutions. (1) The full crossing has \(3\times2\times2=12\) scenarios. (2) Neither sample size nor feature dimension changes alone, so the contrast supports only a comparison of the bundled conditions; it cannot isolate either factor. (3) Taking the second quality level minus the first gives \(0.12-0.20=-0.08\) at \(n=100\), and \(0.10-0.15=-0.05\) at \(n=400\). The contrasts differ by 0.03, so the quality difference is not constant across these sample sizes on the error-rate difference scale. (4) Pairwise balance can show that each pair of factors is represented evenly across its level combinations. It does not, by itself, guarantee that all higher-order interactions are estimable or that a reduced design can support every desired conclusion.