Why Compare Designs on More Than One Criterion?
An adaptive trial design is rarely judged by one operating characteristic. A design may have higher power but require more participants, take longer to complete, or have greater risk of operational failure. Another may control type I error more closely while sacrificing some power. A single ranking can conceal these differences if it combines them into one score.
Pareto comparison offers a way to identify clear inferiority without deciding in advance how much one objective is worth relative to another. It can remove candidates that are no better on any criterion and worse on at least one, while retaining designs whose advantages and disadvantages genuinely conflict. The result is a set of candidates for deliberation, not an automatic recommendation.
The comparison is meaningful only when the objectives have been defined consistently. For example, power should refer to a specified alternative scenario, and type I error should refer to a specified null scenario or a clearly defined summary across null scenarios. Candidate designs must be evaluated using the same scenario definitions, endpoint rules, and simulation assumptions. Otherwise, an apparent design difference may instead reflect a change in what was measured.
Objective Vectors and Dominance
“Better” depends on the direction of an objective. Higher power is generally preferable, while lower type I error, sample size, duration, and failure risk are generally preferable. One can compare values using those directions directly or transform them to a common direction. For clarity, the examples below keep the original units and state the preferred direction for each measure.
This is a partial ordering, not a complete ranking. Two candidates can be incomparable even when their objective values are known exactly. Also, ties do not count as strict improvement: if two designs match on every objective, neither dominates the other under this definition. The frontier depends on the candidates and objectives included. Adding or removing an objective, changing its definition, or adding a new candidate can change which designs are on the frontier.
Worked Example: Finding the Frontier
Suppose a hypothetical simulation compares four candidate designs under the same scenario grid and operational assumptions. Power is to be maximized; all other measures are to be minimized. Sample size is the expected number of participants, duration is in months, and failure risk is the simulated probability of an operational failure.
| Design | Power | Type I error | Expected sample size | Duration (months) | Failure risk |
|---|---|---|---|---|---|
| A | 0.90 | 0.050 | 520 | 18 | 0.040 |
| B | 0.88 | 0.048 | 460 | 15 | 0.030 |
| C | 0.92 | 0.060 | 500 | 16 | 0.050 |
| D | 0.86 | 0.055 | 540 | 20 | 0.060 |
Compare B with D. B has higher power: \(0.88 > 0.86\). It also has lower type I error, expected sample size, duration, and failure risk: \(0.048 < 0.055\), \(460 < 540\), \(15 < 20\), and \(0.030 < 0.060\). Each comparison favors B, with strict improvement on every measure, so B dominates D.
A and B are not comparable by dominance. A has higher power, \(0.90 > 0.88\), but B has lower type I error, a smaller expected sample size, shorter duration, and lower failure risk. C has the highest power, but its type I error and failure risk exceed A’s, so C does not dominate A. A does not dominate C because its power is lower and its expected sample size and duration are higher. The same kind of tradeoff prevents B or C from dominating the other.
Thus A, B, and C are on the frontier among these four candidates, while D is not. This conclusion does not say which frontier design should be chosen. It says only that none of A, B, and C is uniformly at least as good as another on all five listed objectives.
Why Dominated Designs Can Be Set Aside
Dominance has a useful justification. Suppose a decision-maker’s preference is monotone in the stated objectives: improving an objective while holding all others fixed cannot make a design less preferred. If A dominates B, moving from B to A does not worsen any listed objective and improves at least one. Under that preference, B cannot be preferable to A on the listed objectives alone.
The argument is direct: begin with B and replace its value on each objective by A’s value. No replacement worsens the objective, by the dominance conditions. At least one replacement improves it strictly. A monotone preference therefore ranks A at least as highly as B, and, when strict improvement matters, ranks A above B. This justification is conditional on the objective list and preference assumption. If an omitted consideration matters—for example, a regulatory or operational requirement—the comparison must include it or treat it separately.
Dominance is therefore a screening result, not a full decision rule. A nondominated design may still be impractical or unacceptable for reasons not represented in the objective vector. Likewise, a dominated design can appear attractive if the decision-maker values an omitted feature. The objective list should be explicit enough that the frontier answers a useful question, but it should not pretend to represent every governance consideration automatically.
Tradeoffs, Objective Definitions, and Units
A frontier makes competing advantages visible. A committee might prefer greater power, but the Pareto comparison does not specify how much additional sample size is worth a particular increase in power. Nor does it imply that a small numerical change in one measure is equivalent to a similarly sized change in another. Power is dimensionless, sample size is measured in participants, and duration is measured in time; their scales cannot be added meaningfully without additional decision choices.
Worked Example: A Tradeoff Is Not a Dominance Claim
Consider two hypothetical designs compared using power and expected sample size only. Design E has power \(0.91\) and expected sample size 540 participants. Design F has power \(0.88\) and expected sample size 470 participants. Power is to be maximized and sample size minimized.
E has higher power, since \(0.91 > 0.88\), but requires more participants, since \(540 > 470\). F uses fewer participants but has lower power. Neither design is at least as good on both objectives, so neither dominates the other.
It would be possible to assign a numerical value to one power percentage point and one participant, then compute a combined score. But that would add a preference rule, not follow from the simulation results. Without such a rule, the accurate conclusion is that E and F present a tradeoff. Their relative appeal depends on the priorities and constraints applied in the design decision.
Objective definitions also matter. “Power” needs a specified treatment effect and analysis rule; “type I error” needs a specified null scenario or summary; and “duration” needs a defined start and end point. If power is evaluated in one scenario for one design and a different scenario for another, the comparison is not like-for-like. The earlier material on scenario weighting and average performance is relevant when a summary across scenarios is desired; a Pareto analysis can also retain scenario-specific measures rather than compressing them into an average.
Including too many poorly chosen measures can make nearly every candidate appear nondominated. Including too few can make the frontier misleadingly narrow. It is useful to explain why each objective is decision-relevant, state its direction, and report its units or interpretation. If a hard governance requirement is applied, record it distinctly from a preference objective; Pareto dominance itself does not determine the requirement.
Simulation Uncertainty and Diagnostic Checks
In a simulation-based comparison, objective values are estimates. Monte Carlo error can make small differences look like real design advantages. A point-estimate comparison is therefore not automatically reliable evidence that one design dominates another, especially when the apparent improvement is small. The earlier material on Monte Carlo error provides the basis for quantifying this simulation uncertainty.
Worked Example: Checking a Small Simulated Power Difference
Two hypothetical candidates are each evaluated in 10,000 independent simulation replicates under the same alternative scenario. Candidate G is estimated to have power \(0.900\); candidate H has estimated power \(0.895\). For a simulated proportion \(\hat{p}\) based on \(n\) independent replicates, the estimated Monte Carlo standard error is \(SE=\sqrt{\hat{p}(1-\hat{p})/n}\).
For G,
For H,
The estimated power difference is \(0.900-0.895=0.005\), or half a percentage point. Approximate 95% Monte Carlo intervals using estimate \(\pm 1.96SE\) are about \(0.894\) to \(0.906\) for G and \(0.889\) to \(0.901\) for H. These intervals overlap substantially. This calculation does not prove the true powers are equal, and the intervals are not intervals for the design’s performance uncertainty across possible trial settings. It shows that the simulated difference is small relative to the Monte Carlo uncertainty shown by these estimates. A dominance conclusion based on a tiny estimated advantage should be treated as uncertain and investigated with adequate simulation precision.
Several checks help identify a misleading frontier:
- Verify directions. Accidentally maximizing type I error or minimizing power reverses the intended comparison.
- Verify comparability. Confirm that candidates use the same scenario definitions, endpoint, and operating assumptions for each reported objective.
- Inspect scale and definition. A duration measure in weeks for one candidate and months for another, or a differently defined failure event, invalidates a direct numerical comparison.
- Check simulation precision. If rankings change across repeated simulation runs or differences are comparable to Monte Carlo error, the apparent frontier may be unstable.
- Check omitted objectives. A design can be nondominated only relative to the criteria actually included. Record important operational or scientific considerations that are absent from the vector.
How Pareto Comparison Differs From Other Approaches
| Approach | What it provides | Main limitation |
|---|---|---|
| Pareto dominance | Removes candidates that are no better on any listed objective and worse on at least one. | Usually leaves several candidates and does not rank them all. |
| Weighted composite score | Produces a single ranking after assigning weights and combining measures. | Results depend on the chosen scales, transformations, and weights; the tradeoffs may be obscured. |
| Lexicographic ordering | Ranks candidates by a declared priority order, using later objectives to break ties. | Small differences in the highest-priority objective can outweigh large differences elsewhere. |
| Separate requirement checks | Identifies whether candidates meet individually specified conditions. | By itself, it does not rank candidates that meet the conditions or resolve tradeoffs among them. |
These approaches answer different questions and can be used in sequence if their roles are explicit. Pareto comparison is especially useful before a preference-based ranking: it identifies candidates that need not be considered further under the stated objectives, while preserving the unresolved tradeoffs among the remaining designs. It does not replace the specification of requirements or the deliberation needed to select among frontier candidates.
Common Pitfall
A second mistake is to call every design with one favorable result “dominant.” A design dominates another only when it is no worse on every listed objective and strictly better on at least one. A high-power candidate that also has higher type I error or greater failure risk may be an important frontier option, but it is not dominant on that objective set.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Power is to be maximized; expected sample size and duration are to be minimized. Design J has power 0.89, sample size 480, and duration 16 months. Design K has power 0.91, sample size 500, and duration 15 months. Does either dominate the other?
- Design L is no worse than design M on all four listed objectives and is strictly better on two. What can be concluded about M, and what cannot be concluded?
- Why can adding an objective change which candidates are on the Pareto frontier?
- A simulated power estimate is 0.80 from 2,500 independent replicates. Calculate its Monte Carlo standard error using \(SE=\sqrt{\hat{p}(1-\hat{p})/n}\).
Solutions. (1) Neither dominates. K has higher power and shorter duration, but its expected sample size is larger, so each candidate is better on at least one measure and worse on another. (2) L dominates M. M can be removed from the frontier for this objective set under monotone preferences, but the result does not identify the best design overall or account for omitted objectives and requirements. (3) Dominance is defined using every listed objective. A candidate that was no worse on the original objectives may be worse on the newly added objective, so a prior dominance relation can disappear and the frontier can change. (4) Substitute \(\hat{p}=0.80\) and \(n=2500\): \(SE=\sqrt{0.80(0.20)/2500}=\sqrt{0.000064}=0.00800\). The Monte Carlo standard error is 0.00800, or 0.8 percentage points.