Choosing Benchmark Estimands

Why the Estimand Comes Before the Simulation

A simulation benchmark can generate many data sets and return a table of scores, but that table does not by itself say what the comparison is intended to estimate. A method may perform well in common settings and poorly in rare ones; another may be more consistent but never best. An overall score depends on which settings count, how much each counts, and what “performance” means. Choosing the estimand makes these choices explicit before results are examined.

For a multi-omics comparison, a scenario might represent a setting with particular sample sizes, signal patterns, modality coverage, or cohort structure. The estimand is not simply “the average over the simulations.” It is a precisely defined target, such as expected area under the curve for a specified population of scenarios, or the probability that one method has the highest scenario-specific expected performance. The introductory material on formulating a multi-omics benchmark question motivates the comparison; here the focus is on defining its target quantitatively.

The expectation over \(Y_m\) accounts for variation in generated data and any randomized steps in a method. The expectation over \(s\) accounts for variation in the scenario population. These are distinct sources of variation. A benchmark may target a finite, explicitly weighted collection of scenarios, or a broader conceptual distribution \(Q\). In the finite case, write its scenario weights as \(q_s\), with \(q_s\geq 0\) and \(\sum_s q_s=1\). Then \(\mu_m=\sum_s q_s\theta_m(s)\).

A scenario’s weight is not automatically its frequency in a convenient simulation grid. Equal weighting of scenarios gives equal importance to every listed scenario, even if some represent unusual conditions. Weighting by a target-use distribution answers a different question: expected performance when scenarios occur according to that distribution. Both can be legitimate, but they are not interchangeable.

Expected Performance and Pairwise Differences

Expected performance is useful when the scientific question concerns average quality over a declared population of scenarios. State the performance metric and its direction. For example, area under the receiver operating characteristic curve is bounded between 0 and 1, with larger values generally indicating better discrimination. A loss such as mean squared error has the opposite direction: smaller values are better. “Expected performance” without naming the metric and its direction is incomplete.

A pairwise difference summarizes the contrast between two methods on the chosen scale. For methods \(A\) and \(B\), define \(\Delta_{A,B}=\mu_A-\mu_B\). If larger values of \(Y\) indicate better performance, a positive \(\Delta_{A,B}\) favors \(A\) on average under \(Q\). If the metric is a loss, one can either retain this difference and explain that a negative value favors \(A\), or define the contrast in the opposite order. The direction must be declared and used consistently.

A marginal difference can conceal important variation: \(\theta_A(s)-\theta_B(s)\) may be positive in some scenarios and negative in others. The marginal contrast is still well-defined, but it should not be described as universal superiority. Scenario-specific results or a separate estimand for heterogeneity may be needed to show where the methods differ.

Ranking Probabilities Are Separate Targets

A ranking probability answers a question about which method is best across the target scenario population. One useful definition compares the scenario-specific expected performances \(\theta_m(s)\), not the noisy score from one run. For method \(m\), define its strict-winner probability as \(P_{s\sim Q}\{\theta_m(s)>\theta_j(s)\ \text{for every }j\ne m\}\). If ties are possible, the estimand must say how they are treated. For example, one can report the probability of being among the tied best methods, or assign tied methods equal fractional credit so that the winner credits sum to one in each scenario.

This is different from the probability that \(m\) wins on a randomly generated replicate. A replicate-level win event depends on data-generation noise and algorithmic randomness as well as the scenario. A method can have the highest expected performance in a scenario yet lose many individual replicates because its results are variable. Thus, “probability of ranking first” needs to identify whether ranking uses scenario-specific expected performance or observed replicate scores.

Computational Cost and the Target Population

Computational cost should be treated as an outcome with its own target definition. Possible outcomes include wall-clock runtime in seconds, peak memory in gigabytes, or a declared resource measure combining compute time and memory. Specify the computing environment and what operations are included: for example, whether preprocessing, tuning, and data transfer count. Runtime comparisons on different hardware or under different resource limits may not estimate a meaningful method contrast.

For a nonnegative cost \(C_m\), expected cost under the scenario population is \(\kappa_m=E_{s\sim Q}\{E(C_m\mid s)\}\). This answers an average-cost question. If deployment is constrained by a maximum runtime, an average alone may be insufficient; the probability of exceeding the limit, or a high cost quantile, may be more relevant. Likewise, averaging raw runtimes answers a different question from averaging log runtimes. The scale and target must match the practical decision.

Estimation and a Short Justification

In a finite benchmark target, let \(Y_{msr}\) be the performance of method \(m\) in scenario \(s\) on replicate \(r\), and let there be \(R_s\) replicates for scenario \(s\). The scenario mean is \(\bar{Y}_{ms}=R_s^{-1}\sum_{r=1}^{R_s}Y_{msr}\). A direct estimator of the weighted marginal target is \(\hat{\mu}_m=\sum_s q_s\bar{Y}_{ms}\). The weights must be the target weights, not weights that arise accidentally from unequal replicate counts.

To justify this estimator, assume that within each scenario the replicates are generated according to the specified data-generating and analysis procedures, so \(E(Y_{msr}\mid s)=\theta_m(s)\). Then linearity of expectation gives

Thus the estimator is unbiased for the finite weighted target under those conditions. If replicates are independent within and across scenarios, with scenario-specific variance \(\sigma^2_{ms}\), then \( \operatorname{Var}(\hat{\mu}_m)=\sum_s q_s^2\sigma^2_{ms}/R_s. \) This formula shows that more replicates reduce simulation error, while the target weights determine how much each scenario contributes. Correlated replicates or shared random numbers require a variance calculation that accounts for that dependence. Pairing methods on the same generated data often improves precision for a difference, but it does not redefine the target contrast.

For a pairwise contrast, calculate each replicate-level difference on matched runs when appropriate, or subtract the two estimated marginal means. Both target \(\Delta_{A,B}\), provided the same scenario weights and performance definition are used. For a ranking probability based on scenario-specific expected performance, estimate the scenario means first and then rank them; finite-replicate rankings can be unstable when methods are close. The estimator’s uncertainty should reflect this estimation step rather than treating estimated ranks as known.

Choosing Among Alternative Summaries

No single summary answers every comparison question. Alternatives should be described as different estimands, not as interchangeable ways to present the same result.

A practical report may present more than one target: for example, average discrimination, probability of being best, and expected runtime. They should remain distinct. A method can have the highest expected score but rarely rank first, or be best in many scenarios by small margins while having a lower overall mean. A multidimensional decision may be needed when accuracy and cost both matter; reducing them to one number requires an explicitly justified trade-off.

Diagnostics, Failure Modes, and Limits

Before interpreting estimates, check whether the implemented calculation corresponds to the declared target. Confirm that weights are nonnegative and sum to one, that all methods use the same scenario definitions and performance metric, and that scenario-level replication is adequate for estimating means and rankings. Inspect scenario-specific estimates and uncertainty: a plausible weighted average can mask a data or implementation error in one scenario.

• Unequal replicate counts treated as scenario weights. Pooling all runs gives more influence to scenarios with more replicates. If that is not the intended target, compute scenario means and apply the stated \(q_s\).
• Ambiguous ranking event. A first-place rate based on noisy replicate scores is not the probability of having the highest scenario-specific expected performance. Name the event being counted.
• Unspecified ties or failures. State how exact ties, nonconvergence, and undefined metric values are handled. Excluding failed runs can make a method appear better if failures are informative; coding all failures as the same poor score also requires justification.
• Unrepresentative or narrow scenario population. A precise estimate for a chosen grid does not establish performance in scenarios the grid does not represent. The target population limits the scope of the conclusion.
• Cost measured inconsistently. Different hardware, stopping rules, or included computational steps can confound runtime comparisons. Standardize or explicitly define these conditions.

A common mistake is to report “the average performance across all simulations” after generating unequal numbers of replicates per scenario, without specifying the intended weighting. This implicitly gives scenario \(s\) weight proportional to its replicate count, not necessarily its intended importance. The correction is to define \(Q\) first, estimate each scenario’s performance, and combine those estimates using the declared target weights. More replication can improve precision, but it should not silently change the estimand.