Why an Aggregate Ranking Needs an Audit
A simulation benchmark can compare methods across scenarios that differ in sample size, signal strength, missingness, or other design features. A single score is convenient, but it compresses those differences. The resulting ranking can change when the benchmark changes how often scenarios count, how performance metrics are scaled, or whether the aggregation uses scores or ranks. These choices are not merely computational: they determine what “best overall” means.
The earlier material on scenario fidelity concerns whether the simulated scenarios adequately represent the intended data-generating conditions. Here the question is different: given a set of scenarios, does the summary ranking accurately reflect the priorities the benchmark is intended to represent? A defensible summary requires explicit weights, comparable metric scales, and scenario-level results that make trade-offs visible.
This definition assumes that each \(L_{sm}\) measures the same quantity on a comparable scale across scenarios. The weights must represent a stated target distribution or a deliberate policy choice. Equal weights give each listed scenario equal influence, not necessarily each scientifically plausible operating condition equal influence. If the benchmark has multiple metrics, their directions and scales also need to be made comparable before combining them.
Scenario Weights Can Reverse a Ranking
Weights express which scenarios matter and how much. Equal weighting is one possible choice, not a neutral default. A benchmark might instead weight scenarios according to a target population of applications, or might assign separate weights to clinically important operating conditions. The rationale should be specified before examining which method benefits from it.
Worked Example: Changing Scenario Weights
Suppose two hypothetical methods are evaluated using the same loss metric in two scenarios. In Scenario 1, method A has loss 1 and method B has loss 5. In Scenario 2, A has loss 9 and B has loss 6. Lower loss is better. First use equal scenario weights.
For each method, the equal-weight aggregate is the arithmetic mean of its two scenario losses:
Under equal weights, A has the lower aggregate loss by \(5.5-5=0.5\) loss units. Now let \(w\) be the weight on Scenario 1, so Scenario 2 has weight \(1-w\). Then
A is preferred when \(9-8w<6-w\). Rearranging gives \(3<7w\), or \(w>3/7\). At \(w=3/7\), the aggregate losses tie. When \(w<3/7\), B is preferred. Thus, if a target use case gives Scenario 2 more than \(4/7\) of the weight, B ranks ahead of A. The data have not changed; the target represented by the weights has.
This weighting is interpretable only if both scenarios’ losses are defined comparably and the chosen weights have a defensible meaning. Reporting only the overall winner would hide that A is better in Scenario 1 and B is better in Scenario 2.
In general, for two methods A and B, the weighted difference is
A ranking reverses as weights change when scenario-specific differences have opposing signs. If A has lower loss in every scenario, any nonnegative weights summing to one preserve A’s advantage. If some scenarios favor A and others favor B, the aggregate depends on how much weight is assigned to each. This is why a weight-sensitivity analysis is useful when no single target distribution is clearly justified.
Metric Scaling and Direction
A second problem arises when a benchmark combines different metrics, such as predictive error and runtime. Even if both are losses to minimize, a value of 20 milliseconds cannot be added meaningfully to an error of 0.2 without a defined conversion or utility rule. Changing units can change a raw sum: converting seconds to milliseconds multiplies runtime values by 1,000, even though method performance has not changed.
A common approach is to transform each metric to a dimensionless, directionally consistent score before combining metrics. For example, if \(a_k\) and \(b_k\) are prespecified lower and upper reference values for metric \(k\), a scaled loss can be defined as
This transformation preserves the lower-is-better direction and expresses performance relative to the stated reference range. The anchors \(a_k\) and \(b_k\) must be meaningful and fixed consistently; using the observed minimum and maximum from a changing set of methods can make scores depend on which methods happen to be included. A metric that is naturally higher-is-better must first be converted to a loss, or otherwise transformed so that directions agree. There is no universally correct scaling rule: the rule should match the benchmark’s interpretation and be reported.
With scaled losses, one possible summary first combines metrics within each scenario and then scenarios:
The metric weights \(\lambda_k\) and scenario weights \(\pi_s\) answer different questions. The first specifies the relative importance of performance criteria; the second specifies the relative importance of scenarios. Keeping them separate makes the choices easier to interpret and audit.
Worked Example: A Ranking Changed by Metric Scaling
Consider two hypothetical methods evaluated on prediction error (lower is better) and runtime in seconds (lower is better). A has prediction error 0.10 and runtime 20 seconds; B has prediction error 0.12 and runtime 10 seconds. Assume the benchmark has explicitly chosen reference ranges of 0.10 to 0.20 for error and 0 to 100 seconds for runtime, and equal metric weights.
The raw sums would be \(0.10+20=20.10\) for A and \(0.12+10=10.12\) for B. These sums are not meaningful comparisons: the numeric contribution of seconds is arbitrary relative to prediction error, and changing runtime units would change the answer.
Apply the stated range scaling. For prediction error, \(a=0.10\) and \(b=0.20\), so the scaled losses are
For runtime, \(a=0\) and \(b=100\) seconds:
The equal-weight scaled scores are \(L_A^{*}=(0+0.20)/2=0.10\) and \(L_B^{*}=(0.20+0.10)/2=0.15\). Under this particular scaling and equal metric weights, A ranks ahead of B. The conclusion is conditional on the chosen reference ranges and equal importance for the two metrics; it is not an intrinsic, scale-free verdict. The benchmark should show each metric as well as the combined score.
Rank Aggregation and Lost Magnitude
An alternative to aggregating numerical scores is to rank methods separately within each scenario and then aggregate those ranks. Define \(r_{sm}=1\) for the lowest-loss method in scenario \(s\), \(r_{sm}=2\) for the next-lowest, and so on; ties require a stated convention, such as assigning the tied methods the average of the ranks they occupy. The mean rank is
Lower mean rank is preferred. Ranks avoid adding metrics with incompatible units, but they discard the magnitude of the differences. A tiny advantage and a very large advantage both count as a one-place rank difference. Mean ranks can therefore disagree with aggregates of comparable losses. Rank aggregation is most interpretable when relative ordering is the target and the scenarios are weighted deliberately; it is not a substitute for putting meaningful performance differences on a common scale.
Worked Example: Mean Ranks Can Hide Large Differences
Suppose three hypothetical methods have losses in three scenarios as shown below. Lower loss is better, and each scenario receives equal weight.
| Scenario | Method A loss | Method B loss | Method C loss | Ranks A, B, C |
|---|---|---|---|---|
| 1 | 1 | 2 | 100 | 1, 2, 3 |
| 2 | 1 | 2 | 100 | 1, 2, 3 |
| 3 | 1000 | 10 | 0 | 3, 2, 1 |
The mean ranks are
Mean rank therefore puts A first. But the mean losses are
The loss average puts B first by a wide margin. The two summaries answer different questions: mean rank emphasizes ordering across scenarios, while mean loss retains the size of the losses. In this example, A’s severe loss in Scenario 3 matters greatly to the loss average but only changes its rank by one place in the rank summary. Neither summary should be presented as if it were the only possible definition of overall performance.
Assumptions, Diagnostics, and Reporting
Before using an aggregate ranking, check the conditions that give it a coherent interpretation:
- Consistent direction: all component scores are defined so that the preferred direction is clear, typically lower loss.
- Comparable quantities: a score averaged across scenarios measures the same performance construct in each scenario, or an explicit transformation makes the comparison interpretable.
- Defensible weights: scenario and metric weights are stated, nonnegative, sum to one within their respective aggregation, and reflect the benchmark’s intended target or priorities.
- Relevant scenario set: the scenarios represent the conditions to which the summary is meant to apply. Weighting cannot repair important missing scenarios.
- Transparent rank rules: if ranks are used, tie handling and any weighting of scenarios are specified.
Useful diagnostics include a table or plot of scenario-specific performance, a comparison of rankings under plausible alternative weights, and a comparison of conclusions under a justified scaled-score summary and a rank summary. A ranking that changes under small, reasonable weight adjustments is sensitive; report that sensitivity rather than selecting the weights that produce a preferred winner. Inspect whether one scenario or metric dominates the aggregate, and check that scaling anchors are fixed and scientifically interpretable.
The main alternatives have different strengths. A weighted average of comparable losses preserves magnitude and has a direct target-distribution interpretation. A weighted mean rank is less sensitive to extreme numeric values and unit choices, but loses magnitude information. Reporting a vector of scenario-specific scores avoids imposing a single ordering, but is less concise and may leave trade-offs unresolved. A useful benchmark can present a summary while retaining the scenario-level table, so readers can see both the headline and the exceptions.
An aggregate score is a descriptive summary of the specified benchmark and its weighting rules. It is not a universal property of a method, and it does not establish that the top-ranked method will be best in every future application. The defensible conclusion is conditional: under these scenarios, metrics, transformations, and weights, one method has the preferred summary score.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two methods are compared in two equally weighted scenarios. A has losses 2 and 8; B has losses 5 and 6. Which method has the lower mean loss, and what scenario-specific trade-off should also be reported?
- Why can adding prediction error to runtime produce a meaningless aggregate even if both metrics are losses to minimize?
- What information is lost when methods are ranked within each scenario before their ranks are averaged?
- If one method has lower loss than another in every scenario, can nonnegative scenario weights summing to one reverse their weighted-mean ordering? Explain.
Solutions. 1. A’s mean loss is \((2+8)/2=5\); B’s is \((5+6)/2=5.5\), so A ranks first by equal-weight mean loss. A is better in Scenario 1, while B is better in Scenario 2, which should be reported. 2. The metrics have different units and scales; changing the units of runtime can change a raw sum without changing performance. A justified transformation and metric weighting are needed. 3. Ranks discard the magnitude of performance differences: a slight and a very large advantage can contribute the same rank difference. 4. No. If A has lower loss in every scenario, then each difference \(L_{sA}-L_{sB}\) is negative; a weighted sum of those differences with nonnegative weights summing to one remains negative.