Worst-Case Operating Characteristics

Why Examine the Worst Case?

A design can perform well under a central set of assumptions yet behave poorly under a plausible stress condition. In an adaptive trial, for example, power may fall when recruitment is slower than planned, or the probability of a false positive may rise under a particular nuisance-parameter combination. A worst-case operating characteristic summarizes the least favorable performance over a specified collection of scenarios. It is a stress-oriented summary, not a prediction of what will happen most often.

The meaning of “worst” depends on the operating characteristic. Higher power is generally favorable, so its worst value is the smallest one. A higher type I error rate is unfavorable, so its worst value is the largest one. If the concern is how many participants the design might require, a larger expected sample size may be the adverse direction. The direction must be chosen from the design question, not from the numerical values alone.

Definition: Let \(\mathcal{S}\) be a specified set of scenarios, and let \(\theta(s)\) be the true operating characteristic of a fixed design under scenario \(s\). If larger values of \(\theta\) are favorable, its worst-case value over \(\mathcal{S}\) is \(\inf_{s\in\mathcal{S}}\theta(s)\). If smaller values are favorable, its worst-case value is \(\sup_{s\in\mathcal{S}}\theta(s)\). A scenario attaining the relevant infimum or supremum is a least favorable scenario for that characteristic. The infimum or supremum need not be attained for an arbitrary scenario set.

A scenario specifies the data-generating conditions used to evaluate the design, such as treatment effects, control response, subgroup prevalence, recruitment pattern, or missingness mechanism. The scenario set \(\mathcal{S}\) must be stated explicitly: a worst-case summary is always relative to that set. For a finite grid, a minimum or maximum exists, so at least one scenario on the grid attains the worst value. If the set is continuous or otherwise infinite, the infimum or supremum is the appropriate general definition; the design may approach a bound without any scenario attaining it.

Orienting the Operating Characteristic

Before computing an extremum, state whether larger or smaller values represent better performance. For power, \(\theta(s)\) is the probability of rejecting \(H_0\) under a specified alternative, and larger values are usually favorable. For type I error, \(\theta(s)\) is the probability of rejecting \(H_0\) under a specified null scenario, and smaller values are favorable. For expected total sample size, the adverse direction is often larger, although a design objective could impose a different resource measure.

The same scenario need not be least favorable for every characteristic. A scenario that produces the lowest power may not produce the largest type I error or the largest expected sample size. Therefore, report the characteristic, its adverse direction, its worst value, and the scenario or scenarios attaining it. Avoid referring to “the worst scenario” without specifying what is being made worst.

Worked Example: Worst-Case Simulated Power

Suppose a hypothetical adaptive design is evaluated under four alternative scenarios. In each scenario, 5,000 simulated trials are generated, and power is estimated by the proportion that reject \(H_0\).

Scenario	Trials rejecting \(H_0\)	Estimated power
A	4,200 of 5,000	0.840
B	3,900 of 5,000	0.780
C	3,600 of 5,000	0.720
D	4,000 of 5,000	0.800

Power is higher when performance is better, so the worst grid value is the minimum:

\min(0.840,0.780,0.720,0.800)=0.720.

Scenario C is the least favorable scenario among these four for power. The design’s estimated power there is 72.0%, meaning that in this simulation it rejected \(H_0\) in 3,600 of 5,000 trials generated under scenario C. This does not establish that power is at least 72.0% for every possible alternative condition; it establishes only the smallest estimate among the four specified and evaluated scenarios.

At scenario C, the binomial Monte Carlo standard error of the estimated power is

SE=\sqrt{\frac{\hat{p}(1-\hat{p})}{R}} =\sqrt{\frac{0.720(1-0.720)}{5{,}000}} =\sqrt{0.00004032} \approx 0.00635,

where \(R=5{,}000\) is the number of simulated trials. This quantifies simulation variability at that scenario; it is not uncertainty about whether scenario C represents the right set of conditions.

Justifying the Extremum Summary

For a finite scenario grid, the relationship between a worst-case summary and a requirement that must hold across every grid point follows directly from the definitions. Suppose larger values are favorable and the requirement is that performance be at least \(c\) in every scenario. Then:

\theta(s)\ge c\ \text{for every }s\in\mathcal{S} \quad\Longleftrightarrow\quad \min_{s\in\mathcal{S}}\theta(s)\ge c.

For the forward direction, every value is at least \(c\), so the smallest value is at least \(c\). For the reverse direction, every value is at least the minimum; if that minimum is at least \(c\), every scenario value is at least \(c\) as well. When smaller values are favorable and every value must be no greater than \(u\), the corresponding statement is:

\theta(s)\le u\ \text{for every }s\in\mathcal{S} \quad\Longleftrightarrow\quad \max_{s\in\mathcal{S}}\theta(s)\le u.

For an infinite scenario set, replace the minimum or maximum by the appropriate infimum or supremum. These statements concern the true values \(\theta(s)\). In practice, simulation estimates those values, and conclusions based on simulated extrema must account for Monte Carlo error and for how completely the scenario set represents the conditions of interest.

Worked Example: Worst-Case Type I Error and Simulation Precision

A hypothetical design is simulated under four null scenarios, with 10,000 trials per scenario. The estimated type I error rates are 0.047, 0.052, 0.061, and 0.049. Because a larger type I error rate is less favorable, the worst estimated value on this grid is 0.061, in the third scenario.

At that scenario, the binomial Monte Carlo standard error is

SE=\sqrt{\frac{0.061(1-0.061)}{10{,}000}} =\sqrt{\frac{0.061(0.939)}{10{,}000}} =\sqrt{0.0000057279} \approx 0.00239.

A simple approximate 95% interval for the rate at this fixed scenario is

0.061\pm 1.96(0.00239) =0.061\pm 0.00468 \approx [0.0563,0.0657].

This interval reflects Monte Carlo uncertainty for the rate at the selected scenario under a normal approximation. It does not, by itself, provide a simultaneous interval for the maximum true rate across all four scenarios: the scenario was selected because its estimate was largest, and the other scenario estimates also have simulation error. Thus 0.061 is the largest estimated rate on the grid, not a precisely known maximum true rate. If a decision depends on whether the worst-case rate is below a bound, the simulation precision and the selection of the maximum need appropriate attention.

When the Scenario Set Is Not Finite

A finite grid makes the computed minimum or maximum easy to identify, but it does not automatically cover every value between grid points. Suppose a nuisance parameter can vary continuously. The largest error rate on a coarse grid may occur at one endpoint even though the true rate peaks between evaluated points. Conversely, a grid point can be least favorable within the grid without being least favorable over the full scientifically relevant range.

For an infinite set, the distinction between an extremum and a bound matters. For example, if \(\mathcal{S}=(0,1)\) and \(\theta(s)=s\), then \(\inf_{s\in\mathcal{S}}\theta(s)=0\), but there is no scenario \(s\) in the set with \(\theta(s)=0\). The infimum is a worst-case bound, but there is no least favorable scenario attaining it. It would be incorrect to report a minimum of zero attained at a scenario in \((0,1)\).

To support a claim beyond a finite grid, investigators need a reason that the grid covers the relevant behavior—for example, a justified monotonicity result, a sufficiently detailed search with refinement near apparent extrema, or a validated analytic bound. Without such support, describe the result as worst among the evaluated scenarios, rather than worst over all possible conditions.

Worked Example: Worst-Case Expected Sample Size

In a hypothetical simulation of an adaptive design, the mean total sample sizes from 4,000 simulated trials are 410, 455, 438, and 472 participants across scenarios A through D. Suppose the resource concern is the largest expected enrollment. The estimated worst value is therefore

\max(410,455,438,472)=472\text{ participants},

attained at scenario D among the four evaluated scenarios. If the standard deviation of the simulated total sample size in scenario D is 86 participants, the Monte Carlo standard error of its estimated mean is

SE=\frac{s}{\sqrt{R}} =\frac{86}{\sqrt{4{,}000}} \approx \frac{86}{63.25} \approx 1.36\text{ participants}.

This SE describes simulation variability in the estimated mean for scenario D. The conclusion is specifically about the largest estimated mean sample size on this grid. It does not say that 472 is a maximum individual trial size; individual simulated trials may use fewer or more participants. Nor does it identify the scenario with the largest type I error or lowest power, which must be examined separately.

Diagnostics, Comparisons, and Limitations

A worst-case result is most useful when its construction is transparent. Record the scenario set, the operating characteristic, the direction considered adverse, the number of simulated trials per scenario, the estimate at each scenario, and the scenario attaining the estimated extremum. Review the scenario-level results rather than reporting only the single extreme value. This can reveal ties, irregular patterns, or a suspicious isolated estimate.

Check Monte Carlo stability. A small difference between the two most extreme estimates may be no larger than their simulation noise. Increase replication or otherwise assess precision before treating their ordering as meaningful.
Check scenario coverage. A grid maximum or minimum is not a guarantee over unexamined conditions. Refine the grid near apparent extrema when interpolation is not justified.
Check the adverse direction. A high power value is not adverse when the concern is power shortfall; a high type I error rate usually is adverse. State the direction before taking an extremum.
Check the definition of the metric. A maximum expected sample size is not the same as a maximum individual sample size, and the scenario worst for one metric need not be worst for another.

A worst-case summary differs from a reference-scenario result, which reports performance under one chosen condition, and from a range, which reports both ends without necessarily designating one as the decision-relevant adverse direction. A single reference scenario can be easy to communicate but may conceal poor performance elsewhere. A worst-case value highlights vulnerability but can be driven by a scenario that is unlikely or only weakly supported. It does not state how often that scenario will occur, and it should not be interpreted as average performance.

The summary also inherits the assumptions built into the scenario set and simulation model. If relevant forms of misspecification or operational disruption are omitted, a favorable worst-case value does not protect against them. If scenarios are included without a defensible rationale, a highly extreme result may be difficult to interpret. Worst-case analysis is therefore a conditional statement: it describes the design’s most adverse performance over the stated set, under the specified simulation model.

Common pitfall: “The smallest estimated power on our grid is 72%, so the design has at least 72% power under all plausible conditions.” This overstates what the calculation establishes. The corrected statement is: “Among the evaluated scenarios, the smallest estimated power was 72% in scenario C.” Extending the claim to all plausible conditions requires adequate scenario coverage and attention to simulation uncertainty.

Key takeaway: Define the scenario set and adverse direction first. Use an infimum or supremum for general scenario sets, call a scenario least favorable only when it attains that bound, and describe a simulated extremum as applying to the scenarios actually evaluated.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A finite grid has estimated power values 0.81, 0.76, and 0.79. What is the estimated worst-case power on the grid, and which scenario is least favorable for power?
A set of null scenarios has type I error estimates 0.043, 0.058, and 0.051. Which estimate is worst, and why is the minimum not the relevant summary?
For \(\mathcal{S}=(0,1)\) and \(\theta(s)=s\), state the infimum and whether a least favorable scenario attains it.
A simulated worst-case rate is based on the largest estimate among many scenarios. Why might a standard error calculated only at that selected scenario be insufficient to characterize uncertainty in the worst-case rate?

Solutions. (1) The minimum is \(\min(0.81,0.76,0.79)=0.76\); the scenario with estimate 0.76 is least favorable among the three because higher power is better. (2) The worst estimate is the maximum, \(0.058\), because a larger type I error is less favorable; the minimum, 0.043, describes the least error estimate and not the worst error. (3) The infimum is 0, since values of \(s\) can approach 0, but no \(s\in(0,1)\) equals 0, so no scenario attains it. (4) The selected scenario is chosen because its noisy estimate is largest. Other scenario estimates also vary by Monte Carlo error, and selection can make the observed maximum especially noisy; a single-scenario SE does not by itself quantify uncertainty in the maximum across the set.