Why Weight Scenarios?
Design assurance simulations often evaluate a trial under several plausible data-generating scenarios. Each scenario may specify an effect size, nuisance parameters, recruitment pattern, or other conditions that influence the trial's operating characteristics. Reporting the results scenario by scenario shows where performance changes. A summary can also be useful, but it needs to state how the scenarios were combined.
A probability-weighted average answers a specific question: if the scenarios represent possible states of the world with stated probabilities, what is the average operating characteristic over that distribution? It is not automatically a replacement for scenario-specific results. In particular, a good average can coexist with poor performance in a scenario that matters.
The weights are not fractions of simulated replicates unless the simulation was deliberately sampled from that scenario distribution. They express the assumed probability of each scenario in the target distribution. A scenario grid used only to explore sensitivity may have no defensible probabilities; in that case, an average may be mathematically calculable but should not be presented as an average over plausible states without explaining the added assumption.
What the Weighted Average Means
Suppose the true data-generating condition is represented by a random scenario \(S\), with \(P(S=s)=w_s\). If \(A\) is the event that an adaptive design rejects the null hypothesis, then the overall rejection probability is found by conditioning on the scenario:
This identity explains why the weighted average is appropriate when the scenario probabilities describe the target uncertainty distribution. For an alternative scenario set, the result is an average power over that distribution. For a set of null scenarios, it is an average type I error over the specified distribution on null conditions. For sample size, the same conditioning argument gives the average expected sample size.
The interpretation depends on the weights. They may come from external data, a prespecified probability model, or a documented expert elicitation. They should be justified before using them to summarize performance. If they are subjective or uncertain, report that fact and examine how the summary changes under other defensible weights.
Worked Example: Weighted Average Power Across Alternatives
In a hypothetical simulation, a design is evaluated under three alternative scenarios. The weights represent a prespecified distribution over the alternatives. Each conditional power estimate is based on 10,000 simulated trials.
| Scenario | Weight \(w_s\) | Rejections out of 10,000 | Estimated power \(\hat{C}_s\) |
|---|---|---|---|
| A: smaller effect | 0.50 | 8,200 | 0.82 |
| B: intermediate effect | 0.30 | 7,400 | 0.74 |
| C: larger nuisance variability | 0.20 | 5,800 | 0.58 |
The weights are nonnegative and sum to one: \(0.50+0.30+0.20=1.00\). The estimated weighted average power is
Thus the estimated probability of rejection, averaged over the stated alternative-scenario distribution, is 0.748, or 74.8%. The equally weighted mean is \((0.82+0.74+0.58)/3=0.7133\), or 71.33%, because it assigns one third of the weight to each scenario rather than using the stated probabilities. The lowest scenario-specific power is 0.58. These summaries answer different questions; the weighted result does not imply that power is at least 74.8% in every scenario.
Derivation and Simulation Estimates
The weighted formula follows directly from conditioning on the scenario. The events \(\{S=s\}\) are mutually exclusive and exhaustive under the chosen scenario model. Therefore, the rejection event \(A\) can be partitioned into the disjoint events \(A\cap\{S=s\}\), giving
Writing \(C_s=P(A\mid S=s)\) and \(w_s=P(S=s)\) yields \(C_{\mathrm{avg}}=\sum_s w_sC_s\). The same argument applies to an outcome \(Y\) with a finite expectation in each scenario: \(E(Y)=\sum_s w_sE(Y\mid S=s)\). This is why scenario-specific expected sample sizes can be averaged with the same form of calculation.
In simulation, the conditional values are estimated. If \(\hat{C}_s\) is the estimated rejection proportion from \(R_s\) independent simulated trials under scenario \(s\), then \(\hat{C}_{\mathrm{avg}}=\sum_s w_s\hat{C}_s\) estimates the weighted average. With independent simulation runs across scenarios, its Monte Carlo variance is approximately
This expression follows because the variance of a simulated rejection proportion in scenario \(s\) is \(C_s(1-C_s)/R_s\), and independent estimates have zero covariance. In practice, \(C_s\) can be replaced by \(\hat{C}_s\) to estimate the variance. This quantifies simulation error conditional on the scenarios and weights; it does not quantify uncertainty about whether the scenario set or weights are appropriate.
Worked Example: Weighted Average Type I Error
Consider a hypothetical design evaluated under three null scenarios, each with 20,000 simulation replicates. The target type I error is 0.05. The assigned probabilities are part of a prespecified model for the null conditions.
| Null scenario | Weight | Estimated type I error |
|---|---|---|
| A | 0.50 | 0.035 |
| B | 0.30 | 0.047 |
| C | 0.20 | 0.066 |
The weighted average is
The estimated average type I error over this null-scenario distribution is 0.0448, below 0.05. However, scenario C has an estimated type I error of 0.066, above 0.05. The weighted average alone therefore does not establish that the design controls type I error at or below 0.05 in every null scenario. If the requirement is a pointwise bound across the specified null scenarios, each scenario-specific result must be assessed against that requirement.
For comparison, the equal-weight mean is \((0.035+0.047+0.066)/3=0.04933\), while the worst-case type I error is 0.066. The first uses a different distribution; the second focuses on the largest conditional error rather than an average.
Comparing Three Summaries
A weighted average, an unweighted mean, and a worst-case value are not competing calculations of the same estimand. They encode different ways of summarizing the scenario set.
| Summary | Calculation | Interpretation and limitation |
|---|---|---|
| Probability-weighted average | \(\sum_s w_sC_s\), with justified weights summing to one | Average over the stated scenario distribution; can conceal poor performance in a low-weight scenario. |
| Unweighted mean | \(\sum_s C_s/S\) | Average giving every listed scenario equal influence; useful as a grid summary, but not an average under a probability model unless equal probabilities are intended. |
| Worst case | \(\min_s C_s\) for a higher-is-better measure such as power; \(\max_s C_s\) for a lower-is-better measure such as type I error or sample size | Highlights the most adverse listed scenario; depends on the scenario set and does not describe how likely that scenario is. |
The direction of a worst-case summary depends on the objective. For power, the adverse value is the minimum; for type I error or resource use, it is typically the maximum. A weighted average of sample size is an average resource requirement under the scenario distribution, whereas the maximum expected sample size describes the most resource-intensive listed scenario. Neither alone describes the full distribution of trial-level sample sizes.
Worked Example: Average Expected Sample Size
A hypothetical adaptive design has these simulated expected total sample sizes under three operational scenarios. The weights represent the assumed probabilities of those conditions.
| Scenario | Weight | Expected total sample size |
|---|---|---|
| A: slower recruitment | 0.20 | 900 participants |
| B: expected recruitment | 0.50 | 720 participants |
| C: faster recruitment | 0.30 | 540 participants |
The probability-weighted expected sample size is
Under the specified distribution, the average expected total enrollment is 702 participants. The equal-weight mean is \((900+720+540)/3=720\) participants, and the largest scenario-specific expected size is 900 participants. The average can help with planning across the assumed scenario distribution, but it is not a guarantee that enrollment will be near 702 in any individual trial.
Diagnostics, Pitfalls, and Limitations
Before reporting a weighted summary, verify that it answers the intended question. Check that the scenario definitions are distinct and sufficiently cover the conditions of interest; that the weights are nonnegative and sum to one; and that the weights have a documented interpretation. Confirm that the conditional operating characteristics use the same endpoint definition, analysis rule, and simulation conventions. A weighted average is not meaningful if, for example, the scenario-specific values refer to different definitions of success.
- Weights do not sum to one. If weights are intended as probabilities, check their total and resolve missing or duplicated probability mass before calculating the average. Do not silently treat arbitrary importance scores as probabilities.
- Weights are chosen to make the result look favorable. Post hoc downweighting of poor-performing scenarios changes the target distribution rather than repairing performance. Report the basis for weights and show scenario-specific results.
- A low average is mistaken for a guarantee. A weighted type I error below a target does not imply that every null scenario meets the target. State whether the requirement concerns an average or each scenario.
- A scenario grid is treated as exhaustive without justification. The average covers only the specified scenarios and their assigned probabilities. Omitted conditions are not represented just because the listed weights sum to one.
- Monte Carlo error is confused with uncertainty about scenario probabilities. More simulation replicates can reduce Monte Carlo error, but cannot establish that chosen weights reflect reality.
A useful report presents the scenario-specific estimates and weights alongside the weighted average, and may include the equal-weight and worst-case summaries for context. If conclusions change materially under reasonable alternative weights, that dependence is important: it indicates that the summary is sensitive to the assumed scenario distribution. Weighted averaging is also limited when the scenario probabilities are poorly known or when decision requirements are explicitly scenario-specific. In those settings, the conditional results remain central.
Common mistake: “The average null rejection rate is below 0.05, so the design has type I error at most 0.05.” This reasoning confuses an average over scenarios with a bound applying to each scenario. The correction is to report the weighted average as an average and separately check every scenario if the design must satisfy a pointwise type I error requirement.
Check Your Understanding
Try each problem, then compare your reasoning with the solutions.
- A design has scenario-specific powers 0.70 and 0.90, with probabilities 0.25 and 0.75. Calculate the weighted average power.
- Three scenarios have weights 0.2, 0.3, and 0.4. What must be checked before these can be used as scenario probabilities?
- A weighted average type I error is below 0.05, but one null scenario has an estimated rate of 0.061. Does the average establish that all scenarios meet a 0.05 requirement?
- When is an unweighted mean more interpretable than a probability-weighted average?
- Explain the difference between Monte Carlo error in a weighted estimate and uncertainty about the scenario weights.
Solutions. (1) \(0.25(0.70)+0.75(0.90)=0.175+0.675=0.850\), or 85.0%. (2) The weights must be nonnegative and sum to one; here they sum to \(0.2+0.3+0.4=0.9\), so 0.1 of probability mass is unassigned and must be resolved. (3) No. The average summarizes the chosen distribution, while the scenario at 0.061 exceeds 0.05; a pointwise requirement requires checking each scenario. (4) It can be useful when the scenarios are a sensitivity grid with no defensible probabilities and the intended summary gives each grid point equal influence. It should not then be described as an average over a probability distribution. (5) Monte Carlo error reflects finite simulation replicates used to estimate conditional operating characteristics. Weight uncertainty concerns whether the probabilities assigned to scenarios are appropriate; increasing replicate counts reduces the former, not the latter.