Why Operating Characteristics Are Conditional
A sample-size re-estimation rule does not produce one fixed sample size for every possible trial. The interim information may lead to an early stop or to one of several planned final sample sizes. To evaluate the design, ask what happens across repeated trials under specified treatment effects and nuisance parameters: how often does it reject the null hypothesis, how many participants does it use on average, and how often does it stop or reach its cap?
These quantities are operating characteristics. They describe the complete adaptive procedure, including both the interim rule and the final analysis. A final test’s nominal significance level alone does not determine the design’s type I error: the probabilities of reaching each branch and the conditional rejection probabilities in those branches also matter.
For type I error, evaluate this rejection probability under a parameter setting satisfying \(H_0\). For power, evaluate it under a specified alternative. Expected sample size is the probability-weighted average of the total sample size in each state. A cap probability is the probability that the design selects its maximum permitted sample size, not the probability of rejecting \(H_0\) at that sample size.
The sums are the law of total probability and the law of total expectation applied to the design’s possible interim states. They are exact when the state probabilities and conditional outcomes are exact; if those quantities are estimated by simulation, the resulting operating characteristics are estimates with Monte Carlo uncertainty.
A Two-Stage Design and Its Assumptions
Consider a design that reviews the trial after \(m\) participants have contributed interim information. It may stop for futility, continue to a smaller total sample size, or continue to a capped total sample size. The interim state probabilities generally depend on the treatment effect and nuisance parameters. The final rejection probability conditional on a state may also depend on them.
A convenient calculation from state probabilities requires a fully specified model for the interim and final data, the re-estimation rule, and the final test. In particular, one must account for dependence between interim information and the final test statistic. Treating \(r_s(\theta)\) as the nominal level of a fixed-sample test is justified only when that conditional probability really has that value. Independence between a separate interim signal and the final test can make such a simplification valid, but it is an assumption to state, not a default property of internal-pilot data.
Worked Example: Calculating Error, Power, and Sample Size
Suppose a hypothetical trial reviews its data after 40 participants. A prespecified interim rule has three possible states: stop for futility, continue to a total of 100 participants, or continue to the cap of 160 participants. Assume the interim state is independent of the final test statistic conditional on the treatment effect. Under \(H_0\), the state probabilities are 0.20, 0.50, and 0.30, respectively. Under a specified alternative, they are 0.10, 0.45, and 0.45.
In either continuation branch, the final analysis is a one-sided \(z\)-test with critical value 1.96, corresponding to a nominal significance level of approximately 0.025. Under \(H_0\), the final statistic has a standard normal distribution in both branches. Under the alternative, its distributions are \(N(2.50,1)\) at \(N=100\) and \(N(3.162,1)\) at \(N=160\). The futility branch makes no rejection. These distributional and independence assumptions let us calculate each branch’s conditional rejection probability directly.
Type I error. The conditional rejection probability in either continuation branch under \(H_0\) is 0.025, while it is zero at futility. Therefore,
The overall type I error is 0.0200, below the nominal conditional level of 0.025 because 20% of trials stop without a rejection. This calculation does not imply that every sample-size adaptation lowers type I error; the result depends on the specified interim and final-data model.
Power. Let \(\Phi\) denote the standard normal cumulative distribution function. For a final statistic distributed as \(N(\lambda,1)\), the one-sided rejection probability is \(1-\Phi(1.96-\lambda)\). At \(N=100\), this is \(1-\Phi(-0.54)=\Phi(0.54)\approx0.7054\). At \(N=160\), it is \(1-\Phi(-1.202)=\Phi(1.202)\approx0.8853\). Thus,
Power at this specified alternative is approximately 71.58%. This is not a probability that the alternative hypothesis is true; it is the probability of rejection under the stated alternative and design assumptions.
Expected total sample size. The state-specific total sample sizes are 40, 100, and 160. Under \(H_0\), the expected sample size is \(0.20(40)+0.50(100)+0.30(160)=106\) participants. Under the alternative, it is \(0.10(40)+0.45(100)+0.45(160)=121\) participants. The expectation differs because the interim-state probabilities differ.
Cap and stopping probabilities. Under the alternative, the probability of reaching the cap is 0.45, and the probability of stopping for futility at the interim is 0.10. Among trials that continue, the probability of reaching the cap is \(0.45/(1-0.10)=0.500\). The 0.45 unconditional cap probability and the 0.500 conditional probability answer different questions.
State-Specific Probabilities Matter
In the preceding example, independence made the conditional null rejection probability the same in both continuation branches. In other designs, the interim state may be associated with the test statistic used at the end. The correct calculation still conditions on states, but the conditional probabilities must reflect that association.
Worked Example: Weighting Conditional Rejection Probabilities
Consider a hypothetical two-state rule with no early stop. State A selects a total of 80 participants; state B selects the cap of 140. A joint model for the interim and final data gives the following probabilities. The conditional rejection probabilities are outputs of that specified model, not assumptions that each branch automatically has the nominal test level.
| Quantity | State A: \(N=80\) | State B: \(N=140\) |
|---|---|---|
| State probability under \(H_0\) | 0.60 | 0.40 |
| Conditional rejection probability under \(H_0\) | 0.010 | 0.060 |
| State probability under the specified alternative | 0.30 | 0.70 |
| Conditional rejection probability under the alternative | 0.30 | 0.65 |
The overall type I error is \(0.60(0.010)+0.40(0.060)=0.0300\). The power at the specified alternative is \(0.30(0.30)+0.70(0.65)=0.545\), or 54.5%. Expected sample size under \(H_0\) is \(0.60(80)+0.40(140)=104\); under the alternative it is \(0.30(80)+0.70(140)=122\). The cap probabilities are 0.40 under \(H_0\) and 0.70 under the alternative.
For example, using the alternative’s state probabilities with the null conditional rejection probabilities would give \(0.30(0.010)+0.70(0.060)=0.0450\), which is not the type I error. The state probabilities and conditional rejection probabilities must both be evaluated under the same parameter setting.
Stopping Probabilities and Sample-Size Summaries
A stopping probability should name both the event and the parameter setting. Examples include the probability of stopping for futility under a specified alternative, the probability of reaching the maximum sample size under \(H_0\), and the probability of continuing given that the trial did not stop. If a rule has several stopping outcomes, report them separately rather than combining them into an ambiguous “stopping rate.”
When total sample size is fixed within each state, expected sample size follows directly from the state probabilities. If enrollment or follow-up makes the number actually observed random, define the relevant quantity carefully: planned sample size and realized number of participants are not necessarily the same. Costs may also depend on calendar time, testing, or treatment allocation, so expected sample size alone is not always a sufficient resource summary.
Evaluation Methods, Diagnostics, and Limits
For a design with a small number of discrete interim states, direct probability enumeration is transparent: specify \(\pi_s(\theta)\), obtain \(r_s(\theta)\) from the joint model, and combine them. Analytic calculations are efficient when those probabilities have tractable forms. Simulation, discussed in the earlier material on simulating sample-size re-estimation rules, is useful when the interim rule or data dependence makes direct calculation difficult. It estimates the same probabilities by generating repeated trials under specified parameter values and applying the complete design to each.
Regardless of method, check that the model reproduces the actual rule: interim timing, any futility condition, the lower and upper sample-size bounds, and the final test. In simulation, also check that the number of simulated trials is sufficient for the precision needed, particularly for small error probabilities. In analytic work, check probability totals and arithmetic. A mismatch between simulated and enumerated results can reveal a coding error, an omitted state, or an incorrect assumption about conditional dependence.
Operating characteristics are conditional on the assumed treatment effect, nuisance parameters, and data-generating model. They do not establish how the design will perform under every possible scenario. A single power value or expected sample size can conceal important trade-offs: two designs can have the same expected sample size but very different probabilities of reaching the cap or stopping early. A full assessment therefore reports the quantities relevant to the design’s goals across clearly stated scenarios.
Common Mistake
A common error is to report the final test’s nominal significance level as the adaptive design’s type I error. That reasoning ignores the chance of each interim state and any association between the state and the final statistic. The correction is to calculate the state probabilities and conditional rejection probabilities under \(H_0\), then sum their products. Only when the model justifies a common conditional rejection probability can the calculation be simplified to that probability times the chance of reaching a rejecting branch.
Include each stopping outcome and each continuation outcome, with the total sample size attached to every state.
Calculate state probabilities and conditional rejection probabilities under the same null or alternative setting.
Use weighted rejection probabilities for type I error or power, weighted sample sizes for expectation, and sum the probabilities of states matching the cap event.
Confirm the treatment of dependence, verify that state probabilities sum to 1, and state whether any stopping probability is conditional or unconditional.
Check Your Understanding
Use the definitions and calculations above to answer these questions. Solutions follow each question.
- A design has two states with probabilities 0.70 and 0.30 under \(H_0\). Their conditional rejection probabilities are 0.02 and 0.08. What is its type I error under this model?
Solution: \(0.70(0.02)+0.30(0.08)=0.014+0.024=0.038\), so the type I error is 0.038. - A design stops at 50 participants with probability 0.15, continues to 100 with probability 0.55, and reaches a cap of 150 with probability 0.30. What are its expected total sample size and cap probability?
Solution: The expectation is \(0.15(50)+0.55(100)+0.30(150)=7.5+55+45=107.5\) participants. The cap probability is 0.30. - If 0.24 of all trials reach the cap and 0.80 of trials continue beyond the interim, what is the probability of reaching the cap conditional on continuing?
Solution: \(0.24/0.80=0.30\), or 30%. - Why is it generally incorrect to use state probabilities under the alternative with conditional rejection probabilities under \(H_0\) to calculate type I error?
Solution: Type I error is defined under \(H_0\). Both the probability of each state and rejection conditional on that state must be evaluated under a null parameter setting before their products are summed.