Design Assurance Questions and Operating Characteristics

From Design Rules to Assurance Questions

An adaptive trial can change its course as accumulating data arrive: it may alter treatment allocation, stop early, or choose a final treatment or dose. Its performance therefore cannot be summarized by a single feature of the protocol. It depends on how the design rules interact with the possible trial data and the assumptions about the population generating those data.

Design assurance begins by asking a specific question about that performance. For example: Under a scenario in which no treatment is effective, how often would the design declare a treatment successful? Or: Under a clinically relevant benefit, how often would it identify an effective treatment? These questions have different answers, even for the same design. A numerical performance measure is meaningful only when its event or target, the scenario, and the design's decision rules are stated.

Write \(\theta\) for a scenario, and let \(D\) denote the fully specified design. Let \(Y\) represent all trial data that could be observed, and let \(g_D(Y)\) represent the design's actions and final conclusions for those data. Probabilities and expectations below are taken over the possible trial data under \(\theta\), with the design rules \(D\) applied. The adaptive nature of the design matters because the allocation, stopping time, or final decision can vary with \(Y\).

Specify the Question Before the Metric

A useful assurance question identifies four elements: the decision or quantity of interest, the event that counts as success or error, the scenario under which performance is evaluated, and the complete design whose performance is being assessed. If any element is left implicit, the resulting number can be ambiguous. In particular, “the design has 80% power” is incomplete without saying what hypothesis or success rule is used and under which alternative scenario.

For a trial with several doses or treatment arms, “success” might mean declaring at least one effective treatment, selecting the best treatment, or recommending a dose that meets a benefit-risk criterion. These are distinct events. Similarly, a null scenario must specify which treatment effects satisfy the null relevant to the error question. A design's error probability can differ across null scenarios, so a single value may not describe all relevant cases.

Error Rates, Power, and Selection

Suppose the design has a prespecified rule for declaring a treatment effective. Under a null scenario in which that declaration is false, the probability of making the declaration is a type I error probability. For a one-treatment hypothesis test, this is commonly written as \(P(\text{reject }H_0\mid H_0\text{ is true})\). In an adaptive, multi-arm design, the event may instead be “declare at least one ineffective treatment effective.” State the event explicitly: it determines what error rate is being measured.

Power is the probability of meeting a specified success rule under a particular alternative scenario. If success means rejecting \(H_0\), then power under \(\theta\) is \(P(\text{reject }H_0\mid\theta)\). Power is not the probability that the treatment works given the observed trial result. Nor does high power necessarily imply high probability of choosing the best arm: rejecting a null and selecting the correct treatment are different goals.

When several treatments are considered, selection accuracy is another distinct operating characteristic. For example, if one treatment truly has the largest effect, the probability that the design selects it measures correct selection under that scenario. This probability may be low even when power to declare at least one treatment effective is high. It may also be inappropriate to define “correct” as selecting the numerically largest effect if the clinical goal is to find any treatment that satisfies a minimum benefit-risk standard.

Bias and Coverage for a Defined Target

An adaptive design may also produce an estimate and an interval for a treatment effect or another clinical quantity. Before evaluating bias or coverage, specify the target parameter, such as the treatment effect for a defined population, and the estimator or interval procedure. Under scenario \(\theta\), let the true target be \(\psi(\theta)\) and the estimate based on trial data be \(\hat{\psi}\). The bias is the expected estimation error:

Bias is measured in the units of \(\psi\). A positive value means the estimator is high on average relative to the target under that scenario; a negative value means it is low on average. Bias is not the error in one particular trial. Also, a small bias does not imply that individual estimates are precise.

If an interval procedure produces endpoints \(L(Y)\) and \(U(Y)\), its coverage probability under \(\theta\) is

Coverage is the probability that the procedure's interval contains the target across repeated trials under the specified scenario. A nominal 95% interval procedure aims for coverage near 0.95 under its stated conditions; it does not mean there is a 95% probability that a particular fixed parameter lies in the particular calculated interval. For adaptive designs, specify whether the interval targets an effect defined before selection or an effect for the selected treatment, and evaluate coverage for that target.

Expected Sample Size and Resource Questions

When a design can stop early or change enrollment, the total number enrolled is a random quantity. Write \(N\) for that total. Its expected sample size under scenario \(\theta\) is \(E_{\theta}(N)\). The expectation averages over the possible trial outcomes and the resulting design actions. It is not necessarily the number enrolled in a typical realized trial, nor does it describe the full distribution of trial sizes.

Expected sample size answers a resource question such as, “How many participants does the design enroll on average under this scenario?” A design with a modest expected sample size could still have a meaningful chance of reaching its maximum enrollment. For staffing, budget, or duration planning, it may therefore be necessary to ask additional questions about the probability of reaching the maximum or the distribution of trial duration. Those are separate operating characteristics.

Justifying the Probability and Expectation Form

These definitions follow from a simple representation. Let \(I(Y)\) equal 1 when a specified event occurs and 0 otherwise. Its expectation is

Thus an event-based assurance question, such as a false declaration or correct selection, is represented by the probability of its indicator being 1. A numerical quantity such as sample size is summarized by its expectation, which averages its possible values weighted by their probabilities. Bias is the expectation of an estimation error, while coverage is an event probability. This distinction explains why no single metric can stand in for every assurance question.

Choosing a Set of Assurance Questions

A design is usually assessed under more than one scenario. A useful plan connects each scenario to a specific concern rather than producing a long list of unlabelled percentages. For example, a review may ask whether the chance of a false efficacy declaration is acceptable under relevant null scenarios, whether the design has adequate power under a clinically meaningful alternative, and whether expected enrollment is feasible under both favorable and unfavorable conditions.

Different approaches can contribute to the assessment. When a simple fixed-sample design and a standard model make the relevant probability tractable, an analytical calculation can provide a direct answer under its assumptions. For complex adaptation, probabilities and expectations can instead be assessed over the full range of trial outcomes implied by the scenario and design. Either way, the interpretation depends on the same essentials: a defined event or target, a scenario, and explicit design rules. An analytical answer is not automatically more credible if its assumptions do not represent the design; a simulation-based answer is not automatically informative if it evaluates the wrong scenario or decision event.

The assumptions behind each scenario deserve scrutiny. They may include outcome distributions, treatment effects, variability, correlations, delayed or missing outcomes, and the relationship between interim information and later outcomes. These are assumptions for evaluating performance, not facts established by the operating characteristic itself. Compare performance across scenarios that represent plausible conditions and important departures from the primary assumptions. An apparently strong result under one convenient scenario cannot establish robustness.

Diagnostics, Limitations, and Common Mistakes

A performance summary is suspect if the event being counted cannot be stated in one sentence, if the target parameter changes between calculations, or if the reported value has no named scenario. Check that outcome categories are mutually exclusive and collectively exhaustive when they are used to calculate probabilities, and that the probabilities sum to one. For bias, check that the estimate and target use the same scale and units. For coverage, verify that the interval endpoints are compared with the intended target. For sample size, verify that the random quantity is total enrollment rather than a fixed planned maximum.

• Mistake: “Power is the probability the null is true.” Power is the probability of meeting a specified success rule under a specified alternative scenario; it is not a probability about whether a hypothesis is true.
• Mistake: “High power means the best treatment will be selected.” Power and selection accuracy refer to different events. State and evaluate the event that corresponds to the clinical decision.
• Mistake: “A 95% interval has a 95% chance of containing the fixed effect.” Coverage describes the repeated-trial performance of an interval procedure under a scenario; it does not assign probability to a fixed parameter after observing one interval.
• Mistake: “The expected sample size is the maximum or the usual realized size.” It is a probability-weighted average. Report additional summaries when the chance of high enrollment matters.
• Mistake: “Meeting one target proves the design is assured.” Error control, power, estimation, selection, and feasibility are different objectives. A design can perform well on one and poorly on another.

Operating characteristics are conditional descriptions, not universal guarantees. They do not establish that a model is correct, that assumptions are plausible, or that the chosen success criterion is clinically appropriate. Their value is that they turn design concerns into explicit, assessable questions and make the assumptions behind the answers visible.