Weighting Scenarios in Overall Design Summaries

Why Scenario Weights Matter

A design can perform well under one set of assumptions and poorly under another. For example, a sampling schedule might estimate a typical patient’s clearance precisely but perform less well when observations are missing or when the model is misspecified. An overall summary can help compare designs across scenarios, but it answers a decision question only after the relative importance of those scenarios has been specified.

The central choice is therefore not merely how to calculate an average. It is which scenarios the average represents, and why each receives its weight. In simulation studies, the number of replicates run for a scenario is a computational choice; it does not, by itself, establish how important that scenario is to the design decision.

Definition: Let $d$ index a candidate design and $s=1,\ldots,S$ index scenarios. Let $\hat L_{ds}$ be the estimated value of a performance measure for design $d$ in scenario $s$, expressed so that smaller values are preferred. Given scenario weights $w_s$ satisfying $w_s\geq 0$ and $\sum_{s=1}^{S}w_s=1$, the weighted overall performance estimate is $$ \hat L_d(w)=\sum_{s=1}^{S}w_s\hat L_{ds}. $$ The weights describe the relative importance or probability assigned to the scenarios for this summary; they are not the number of simulation replicates in each scenario.

The formula applies directly to a measure where lower is better, such as root mean squared error or computational cost, provided the units and interpretation are suitable for averaging. For a measure where higher is better, such as a success probability, the weighted average is still calculable, but the direction of preference must be interpreted accordingly. Alternatively, define a loss that increases as performance worsens. For coverage, for instance, a decision may focus on how far coverage is from a prespecified target rather than treating higher coverage as always better.

Meaning and Assumptions of the Weighted Summary

A useful interpretation follows by treating the scenario as a random variable $S$ with $P(S=s)=w_s$. If $L_d$ denotes the performance loss for design $d$ in the scenario drawn, then the overall expected loss is

E(L_d)=\sum_{s=1}^{S}P(S=s)E(L_d\mid S=s) \approx \sum_{s=1}^{S}w_s\hat L_{ds}.

This interpretation is justified when the scenarios represent the relevant states of the world and the weights represent their probabilities or decision importance. If weights are importance scores rather than probabilities, they should still be normalized to sum to one for this weighted-average calculation, but the result is a decision-weighted summary, not necessarily a population expectation.

Several conditions matter for a defensible summary:

Scenario set: The scenarios should cover the conditions relevant to the intended design decision. A weighted average cannot account for a consequential scenario that was not simulated.
Meaningful weights: Weights should be chosen for a stated reason, such as target-population prevalence, a policy assumption, or a prespecified priority. They should not be selected after seeing results simply to favor a design.
Comparable performance values: The scenario-specific values must measure the same quantity on a scale where averaging is interpretable. Bias in concentration units, computational time, and coverage probability should not be combined directly into one sum.
Consistent estimation: Each $\hat L_{ds}$ should estimate the same defined scenario-specific performance measure, using the specified simulation and analysis pipeline.
Visible scenario results: The weighted estimate supplements, rather than replaces, the scenario-specific results. A low overall loss can conceal an unacceptable loss in one important scenario.

Why the Formula Is an Expectation

The weighted sum is not an arbitrary scoring rule when the weights are probabilities. Partition the possible conditions into mutually exclusive scenarios. The indicator $I(S=s)$ equals 1 when scenario $s$ occurs and 0 otherwise, so the loss can be written as $L_d=\sum_s I(S=s)L_d$. Taking expectations and applying linearity gives

E(L_d) =E\left[\sum_{s=1}^{S}I(S=s)L_d\right] =\sum_{s=1}^{S}E\left[I(S=s)L_d\right] =\sum_{s=1}^{S}P(S=s)E(L_d\mid S=s).

The last equality holds because $I(S=s)$ is zero outside scenario $s$, and within that scenario its probability-weighted contribution is $P(S=s)E(L_d\mid S=s)$. Replacing each conditional expectation by its simulation estimate gives the plug-in estimate $\hat L_d(w)$. This derivation also clarifies the scope of the result: it summarizes expected performance under the chosen scenario distribution, not a guarantee about every scenario.

Worked Example: Weighting Estimation Error Across Scenarios

Suppose two hypothetical sampling designs are assessed using root mean squared error (RMSE) for clearance, in L/h. Three scenarios represent the target use case, with weights $0.50$, $0.30$, and $0.20$. The scenario-specific RMSE estimates are:

Scenario	Weight	Design A RMSE (L/h)	Design B RMSE (L/h)
Typical covariates	0.50	1.8	2.2
High residual error	0.30	2.4	2.6
Misspecified model	0.20	5.0	3.0

The weights are nonnegative and sum to $0.50+0.30+0.20=1.00$. For Design A,

\hat L_A=0.50(1.8)+0.30(2.4)+0.20(5.0) =0.90+0.72+1.00=2.62\ \text{L/h}.

For Design B,

\hat L_B=0.50(2.2)+0.30(2.6)+0.20(3.0) =1.10+0.78+0.60=2.48\ \text{L/h}.

Under these weights and this RMSE criterion, Design B has the lower weighted RMSE by $2.62-2.48=0.14$ L/h. This conclusion does not mean Design B is better in every scenario: Design A has lower RMSE in the first two scenarios, while Design B performs better under misspecification. The overall result depends on the stipulated scenario weights.

Alternative Weights and Design Comparisons

A single set of weights often rests on uncertain assumptions. A useful sensitivity analysis recalculates the summary under alternative plausible weights and checks whether the design comparison changes. The purpose is not to search for a favorable weighting, but to show which assumptions drive the conclusion.

Worked Example: A Design Preference That Reverses

Consider two hypothetical scenarios: a common setting and a rare but challenging setting. Estimated loss is measured on the same scale, with lower values preferred. Design A has losses 2 in the common setting and 10 in the challenging setting; Design B has losses 4 and 5, respectively.

If target-population prevalence is represented by weights $0.80$ and $0.20$, then

\hat L_A=0.80(2)+0.20(10)=1.6+2.0=3.6, \qquad \hat L_B=0.80(4)+0.20(5)=3.2+1.0=4.2.

Design A has the lower weighted loss under this prevalence-based weighting. Now give the two scenarios equal weight:

\hat L_A=0.50(2)+0.50(10)=6.0, \qquad \hat L_B=0.50(4)+0.50(5)=4.5.

With equal weights, Design B has the lower weighted loss. The reversal is informative: Design A’s advantage depends on the common setting receiving greater weight, while Design B is more balanced across these two scenarios. Neither weighting is automatically correct. The relevant choice depends on whether the goal is expected performance under prevalence or a decision that gives the challenging scenario equal importance.

This sensitivity analysis should be presented with its assumptions. “Equal weights” means each listed scenario contributes equally, even if the scenarios differ in prevalence or breadth. Prevalence weights answer a different question. If a scenario represents a broad range of conditions while another is a single extreme setting, assigning both equal weight can also make the result depend on how the scenario set was divided into categories.

Do Not Confuse Scenario Weights With Replicate Counts

Simulation replicates estimate performance within a scenario. Scenario weights combine those scenario-specific estimates for the target summary. These are separate roles. A scenario simulated 10,000 times does not thereby deserve more influence than one simulated 2,000 times. More replicates may make its estimate more precise, but they do not change its substantive importance.

For example, if scenario-specific estimates are $\hat L_{d1}$ and $\hat L_{d2}$, the intended weighted estimate remains $w_1\hat L_{d1}+w_2\hat L_{d2}$, even if the replicate counts differ. Pooling all replicates without accounting for scenarios instead produces an estimate whose effective scenario contribution is determined by the replicate allocation. That may be appropriate only if this allocation was deliberately chosen to represent the target scenario distribution; usually it is chosen for computational precision, so the pooled result can answer the wrong question.

Worked Example: Combining Scenario-Specific Coverage

Suppose a hypothetical design is evaluated for a confidence interval with nominal coverage target $0.90$. Estimated coverage is $0.91$ in a typical scenario and $0.72$ under substantial missingness. The target population assigns weights $0.80$ and $0.20$, respectively. The weighted coverage is

\hat C=0.80(0.91)+0.20(0.72)=0.728+0.144=0.872.

Thus the estimated coverage averaged over the stated scenario distribution is $0.872$, or 87.2%. This is 2.8 percentage points below the nominal 90% target. The scenario-specific value of 72% still matters: a weighted average of 87.2% does not establish acceptable coverage under substantial missingness. If the design requirement is that coverage meet a threshold in every prespecified scenario, this average alone is not the relevant pass/fail criterion; the scenario-specific requirement must also be checked.

Diagnostics, Alternatives, and Limitations

Before interpreting an overall summary, verify the arithmetic and the decision meaning. Confirm that weights are nonnegative and sum to one; if they do not, normalize them only when the underlying relative weights are meaningful and normalization is intended. Confirm that all scenario-specific estimates use the same metric and direction of preference. Then compare the summary with the full scenario table to find whether one scenario dominates the result or is obscured by it.

Common alternatives answer different questions. Equal weighting treats each defined scenario as equally important. Prevalence weighting estimates average performance under a target population distribution when the weights credibly represent that distribution. A worst-case summary emphasizes the largest scenario-specific loss rather than average loss. Reporting all scenario-specific results without aggregation avoids choosing a single weight set, though it may not provide one concise comparison. These approaches are not interchangeable: the appropriate one follows from the design objective and consequences of poor performance.

A weighted average has important limitations. It can mask severe failure in a low-weight scenario, and it is sensitive to how scenarios are specified and weighted. It also cannot repair a poorly chosen performance metric. For instance, averaging bias values with different signs can yield a small result through cancellation even when errors are substantial; a loss such as absolute bias may be more appropriate if the goal is to penalize error magnitude. Nor should distinct metrics be combined merely because weights can be written down: combining RMSE, failure rate, and computation time requires a defined common scale or an explicit decision rule.

A common mistake is to say, “Scenario 1 had four times as many replicates, so it should get four times the weight.” That confuses precision of estimation with importance to the design decision. Replicate counts affect Monte Carlo precision; scenario weights specify the target summary. Another mistake is to report only the weighted result and conclude that the design performs well overall. The correction is to state the metric, weights and rationale, show scenario-specific estimates, and describe the overall conclusion as conditional on those choices.

Key takeaway: A weighted overall performance estimate is meaningful only with an explicit metric, a relevant scenario set, and defensible weights. Treat it as a summary under those assumptions, and retain scenario-specific results so that important weaknesses remain visible.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two scenarios have losses 3 and 7. With weights 0.25 and 0.75, what is the weighted loss?
Why should a scenario with more simulation replicates not automatically receive more weight?
A design has weighted average coverage above its target but coverage below target in one scenario. What can and cannot be concluded?
Two scenario-specific estimates are bias in mg/L and computation time in minutes. Why is their weighted sum not a useful overall performance measure as stated?

Solutions. (1) The weighted loss is $0.25(3)+0.75(7)=0.75+5.25=6.00$. (2) Replicate count determines how much simulation information estimates performance in that scenario; it does not define the scenario’s importance in the target decision. Use the specified scenario weights for the overall summary. (3) One may conclude that average coverage under the stated scenario weights exceeds the target. One may not conclude that coverage meets the target in every scenario; the below-target scenario remains a potential concern and should be reported. (4) The quantities have different units and meanings, so adding their weighted values has no interpretable scale. Analyze them separately or define and justify a common scale or decision rule before combining them.