Selecting Designs With Robust Performance Criteria

Why Design Rankings Depend on the Criterion

A trial design is rarely evaluated under one known data-generating condition. The treatment effect, residual variability, dropout, control response, or other features may be uncertain when a design decision is made. Simulation can estimate how candidate designs perform across a set of plausible conditions, but it does not by itself determine which design is preferable. That requires a criterion for combining the scenario-specific results.

A design with the highest average power may perform poorly in an important unfavorable scenario. A design with the best worst-case power may sacrifice substantial performance in scenarios considered much more likely. A lower-percentile criterion focuses on a selected part of the performance distribution, while a scenario-weighted criterion summarizes performance using specified scenario probabilities. These criteria answer different questions; none is universally best.

Definition: A scenario is a specified set of assumptions about the data-generating process, such as treatment effect, variability, or dropout. An operating characteristic (OC) is a performance quantity for a design under a scenario, such as power, probability of meeting a decision rule, or type I error rate. Write \(OC(d,s)\) for the OC of design \(d\) in scenario \(s\). Candidate designs should be compared using the same scenarios, outcome definition, and analysis rule.

For example, if the OC is power, larger values are favorable. If it is type I error, larger values are generally unfavorable and must be controlled. A robustness criterion must therefore be oriented to the quantity being assessed. The examples below use power as the objective, while treating type I error as a separate constraint when appropriate.

Three Ways to Summarize Scenario Performance

Suppose a finite set of scenarios \(s=1,\ldots,K\) has been specified. For an OC where larger values are preferable, the worst-case value for design \(d\) is the smallest scenario-specific value:

OC_{\mathrm{worst}}(d)=\min_{1\leq s\leq K} OC(d,s).

Selecting the design with the largest worst-case value is a maximin rule: it maximizes the minimum performance across the scenarios included. This guarantee is conditional on that scenario set. It does not protect against a scenario that was omitted.

A percentile criterion summarizes a chosen quantile of scenario performance. With equal scenario weights, sort the \(K\) values from lowest to highest and select a quantile convention in advance. For an empirical lower quantile, one common definition is the ordered value at rank \(\lceil Kq\rceil\), where \(q\) is the chosen probability level. Quantile conventions can differ, especially for small \(K\); the convention should be reported so the calculation is reproducible. A lower percentile does not guarantee performance in every scenario.

A scenario-weighted criterion uses weights \(w_s\) that represent the chosen relative importance or probability assigned to each scenario. The weights must be nonnegative and sum to one:

OC_{\mathrm{weighted}}(d)=\sum_{s=1}^{K} w_s OC(d,s), \qquad w_s\geq 0,\qquad \sum_{s=1}^{K} w_s=1.

When the weights represent probabilities for mutually exclusive scenarios that exhaust the uncertainty being considered, this quantity is the expected OC over that scenario distribution. If weights instead encode policy priorities, the result is a weighted score, not necessarily a probability-based expectation.

Worked Example: Selecting by Worst-Case Power

Suppose two hypothetical designs are simulated under four plausible treatment-effect and dropout combinations. The estimated powers are:

Design	Scenario 1	Scenario 2	Scenario 3	Scenario 4
A	0.80	0.74	0.66	0.58
B	0.75	0.72	0.70	0.68

Under the stated scenario set, Design A has worst-case power \(\min(0.80,0.74,0.66,0.58)=0.58\). Design B has worst-case power \(\min(0.75,0.72,0.70,0.68)=0.68\). Thus a maximin power criterion selects Design B, even though Design A has greater power in the first two scenarios. The conclusion is specifically that B has the higher minimum power among these four scenarios; it is not a claim that B is best under every possible condition.

Worked Example: Selecting by a Lower Percentile

Consider ten hypothetical, equally weighted scenarios. The simulated powers for Design A, already sorted from lowest to highest, are 0.30, 0.74, 0.76, 0.78, 0.80, 0.82, 0.84, 0.86, 0.88, and 0.90. Those for Design B are 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, and 0.81. Use the empirical lower 20th percentile defined as the ordered value at rank \(\lceil Kq\rceil\).

Here \(K=10\) and \(q=0.20\), so the rank is \(\lceil10(0.20)\rceil=2\). Design A's value at rank 2 is 0.74; Design B's is 0.73. The lower-20th-percentile criterion therefore selects A. By contrast, the worst-case powers are 0.30 for A and 0.72 for B, so a worst-case criterion selects B. The apparent disagreement is not a calculation error: the chosen percentile discounts the single very low value for A, whereas the worst-case rule is determined by that value.

Because this small example uses only ten scenarios, its empirical percentile is coarse and sensitive to individual scenario values. In practice, the scenario set and quantile convention should be justified, and the percentile should not be presented as if it described unmodelled conditions.

Worked Example: Selecting by Scenario-Weighted Power

Suppose three hypothetical scenarios have assigned weights 0.50, 0.30, and 0.20. The simulated powers are 0.82, 0.70, and 0.40 for Design C, and 0.75, 0.68, and 0.65 for Design D. The weights sum to \(0.50+0.30+0.20=1.00\).

For C, the weighted power is \((0.50)(0.82)+(0.30)(0.70)+(0.20)(0.40)=0.410+0.210+0.080=0.700\). For D, it is \((0.50)(0.75)+(0.30)(0.68)+(0.20)(0.65)=0.375+0.204+0.130=0.709\). The weighted criterion selects D, by 0.009, or 0.9 percentage points. C has higher power in the first scenario, but D performs better under the specified weighted summary. The ranking depends on the weights: changing them changes the question being answered.

This result is a meaningful expected power only if the weights are defensible probabilities for the scenarios under consideration. If they express stakeholder priorities, report them as priorities and interpret 0.709 as a weighted score rather than an unconditional probability of success.

Constraints, Simulation Error, and a Justification

Robust performance criteria are often used to rank designs that first satisfy minimum requirements. For example, a design might be required to keep the maximum type I error across null scenarios below a prespecified limit, while power is optimized using a worst-case, percentile, or weighted criterion. This separates feasibility from preference: a design that violates a required error-rate constraint is not rescued by high average power.

If the operating characteristic is estimated by \(R\) independent simulation replicates, each replicate producing success or failure, the estimate is a sample proportion \(\hat{p}\). Its approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/R}\). This quantifies simulation noise in the estimated probability, not uncertainty about which scenario is true or whether the scenario set is adequate.

Worked Example: Checking Monte Carlo Precision

In a hypothetical simulation with \(R=5000\) independent replicates, a decision rule succeeds in 235 replicates, giving \(\hat{p}=235/5000=0.0470\). The estimated Monte Carlo standard error is:

SE(\hat{p})=\sqrt{\frac{\hat{p}(1-\hat{p})}{R}} =\sqrt{\frac{0.0470(0.9530)}{5000}} =\sqrt{0.0000089582} \approx 0.002993.

Thus the estimated probability is 4.70%, with Monte Carlo standard error about 0.299 percentage points. If two candidate designs differ by a similarly small amount, their ranking may be sensitive to simulation noise. This standard error does not account for model misspecification, uncertainty in the scenario weights, or omitted scenarios.

The basic rationale for a weighted criterion follows directly from conditioning on the scenario. If scenario \(s\) occurs with probability \(w_s\), and the probability of success conditional on that scenario is \(OC(d,s)\), then the law of total probability gives:

P(\text{success}\mid d) =\sum_{s=1}^{K}P(\text{success}\mid d,s)P(s) =\sum_{s=1}^{K}OC(d,s)w_s.

The derivation requires that the scenarios partition the modeled uncertainty and that the weights correctly represent their probabilities. Without those conditions, the weighted sum remains a useful score if defined as such, but it does not equal an overall success probability.

Diagnostics and Choosing Among Criteria

Before ranking designs, inspect scenario-level results rather than only the summary. A single unusually poor value can dominate a worst-case result; a percentile can conceal a severe failure if that scenario lies below the selected quantile; and a weighted average can mask weak performance in a low-weight scenario. These may be intended properties of the criteria, but they should be visible to decision makers.

Check scenario coverage. Include conditions that are scientifically plausible and decision-relevant. A robust result over an incomplete set is only robust within that set.
Check weights and quantile definitions. Record who or what justifies the weights, and state the quantile convention, particularly when the number of scenarios is small.
Check simulation stability. Use adequate replicates for the precision needed to distinguish designs. Repeat or increase simulation when close rankings are driven by Monte Carlo variation.
Check the whole OC profile. Report scenario-specific power and error rates alongside summaries. Confirm that a favorable power criterion does not hide violation of type I error or other design requirements.
Check implementation and convergence. Confirm that simulated trial data are analyzed as intended and that numerical failures are not silently treated as ordinary outcomes. The earlier material on diagnosing numerical convergence and analysis failures is relevant to this check.

An average across scenarios is the scenario-weighted criterion with equal weights. It is easy to interpret when equal weighting is justified, but it offers no protection against a poor tail scenario. Worst-case selection gives the strongest protection against included scenarios but can be controlled by one extreme or weakly plausible condition. A percentile criterion lies between these approaches: it emphasizes lower-tail performance while allowing a specified fraction of scenarios to fall below the threshold. Scenario weighting can be more directly tied to a probability model or explicit priorities, but the ranking is only as credible as those weights.

No summary replaces judgment about consequences. If failure in one scenario would be unacceptable, use a constraint or a worst-case criterion rather than hoping an average will reveal the problem. If scenarios represent credible but differently likely states of the world, a weighted criterion may be appropriate. If the goal is resilience to a specified fraction of adverse scenarios, a lower percentile may be informative. It is often useful to show more than one criterion as a sensitivity analysis, explain why they differ, and avoid claiming that one design is unambiguously optimal when rankings change with reasonable assumptions.

Common Pitfall

Common mistake: “Design A has the highest average power, so it is the most robust.” An average does not establish robust performance: A may have a low worst-case value, or poor performance in scenarios that matter greatly. Correction: define the target criterion before comparing designs, report the scenario-specific OCs, and describe the scope of any robustness claim. A weighted average supports an expected-performance interpretation only when its weights are valid scenario probabilities.

A second error is treating estimated OCs as exact. Simulation estimates have Monte Carlo error, and small differences in a ranking may not be stable. Report the number of replicates and enough precision to assess that error; do not imply that Monte Carlo standard errors address uncertainty about the underlying clinical assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two designs have scenario-specific powers \((0.78,0.62,0.70)\) and \((0.72,0.68,0.69)\). Which has higher worst-case power?
With weights \((0.60,0.25,0.15)\), powers for a design are \((0.80,0.60,0.40)\). Calculate its weighted power.
Why can the design with the highest lower 20th percentile have a lower worst-case power than another design?
A design has weighted power 0.80 but type I error 0.057 in a null scenario where the maximum allowed rate is 0.05. Should it be selected on the basis of power alone?

Solutions. (1) The first design has minimum power \(\min(0.78,0.62,0.70)=0.62\); the second has \(\min(0.72,0.68,0.69)=0.68\), so the second has higher worst-case power. (2) The weighted power is \((0.60)(0.80)+(0.25)(0.60)+(0.15)(0.40)=0.48+0.15+0.06=0.69\). (3) A lower percentile is determined by an ordered value above the minimum under the stated quantile convention, so a very poor scenario below that rank may affect the worst case without changing the percentile. (4) No. It violates the stated type I error constraint, so it is infeasible under that requirement regardless of its weighted power.