Why Order Outcomes at the Stage of Stopping?
When a trial may stop early, the observed data include both the responses and the rule that determined how many responses were collected. An estimate calculated as if the observed sample size had been fixed in advance can therefore fail to reflect the selection induced by stopping. A stagewise ordering estimate uses the trial’s stopping rule directly: it ranks possible outcomes by the evidence they would provide against a candidate treatment effect at the stage where each outcome occurs.
The ordering must be specified before it can be used. Different stagewise test statistics, tail conventions, or tie rules can rank outcomes differently, and therefore produce different estimates. The examples below use a simple one-sided ordering for a binary treatment response. The construction applies more broadly when a trial has a specified sequence of looks and a specified stagewise test statistic.
The one-half criterion gives a central calibration of the observed outcome’s rank under each candidate value. In a discrete sample space, the tail probability can jump over one-half; an estimate may then require a prespecified convention for ties, randomization, or interpolation. The examples have no such ambiguity at their solutions.
A Sequential Bernoulli Model and Its Ordering
Suppose treated participants have independent binary responses \(Y_i\sim\mathrm{Bernoulli}(p)\), where \(p\) is the treatment-arm response probability. For illustration, let the comparator response probability \(p_C\) be known, so the treatment effect is the risk difference \(\theta=p-p_C\). Treating \(p_C\) as known is a simplifying assumption; if it is estimated from trial data, its uncertainty and the joint allocation and stopping design must be included in the ordering calculation.
Consider a design that examines responses sequentially and stops at the first response or after a prespecified maximum number of participants. Write \(X_n=\sum_{i=1}^nY_i\) for the number of responses among the first \(n\) treated participants. For a candidate null response probability \(p_0\), use the exact one-sided stagewise p-value \(q_n(p_0)=P_{p_0}\{\mathrm{Binomial}(n,p_0)\geq X_n\}\). Smaller values indicate stronger evidence for a response probability above \(p_0\). Outcomes are ranked by these p-values at their respective stopping stages: an outcome is at least as extreme as the observed one when its stagewise p-value is no larger.
This is a design-specific definition, not a universal rule that all stagewise-ordering procedures must use. In a continuous-outcome trial, for example, the stagewise statistic might be a standardized score or a likelihood-ratio statistic. Whichever statistic is chosen, it must be defined for every possible stopping stage, and the probability of the resulting ordering tail must be calculated under the actual stopping rule.
From an Ordering to an Estimate
Let \(A(p_0;x_{\mathrm{obs}})\) denote the outcomes at least as strongly ordered against \(p_0\) as the observed outcome \(x_{\mathrm{obs}}\). The ordering tail can depend on \(p_0\), because stagewise evidence is evaluated against that candidate value. The stagewise ordering estimate \(\hat p_{\mathrm{SO}}\) solves \(P_{\hat p_{\mathrm{SO}}}\{A(\hat p_{\mathrm{SO}};x_{\mathrm{obs}})\}=0.5\). The treatment-effect estimate is then \(\hat\theta_{\mathrm{SO}}=\hat p_{\mathrm{SO}}-p_C\) in the illustrative known-comparator setting.
The probability in this equation is over all possible paths permitted by the trial, including the possibility of stopping at each stage. It is not generally the probability conditional on the observed stopping stage. Replacing it with a conditional probability defines a different procedure and requires an explicit choice of what is being conditioned on.
Worked Example: A Response at the Second Stage
A hypothetical trial stops at the first response or after two treated participants. The observed path is failure followed by response, so \(X_2=1\) and the trial stops at stage two. Suppose the comparator response probability is known to be \(p_C=0.20\). Find the stagewise ordering estimate and the risk-difference estimate.
For a candidate \(p_0<0.5\), the stagewise p-value for a response at stage one is \(q_1(p_0)=p_0\). For one response among two participants, it is \(q_2(p_0)=P_{p_0}\{\mathrm{Binomial}(2,p_0)\geq1\}=1-(1-p_0)^2=2p_0-p_0^2\). Since \(2p_0-p_0^2>p_0\) when \(0<p_0<1\), a response at stage one is ranked as stronger evidence than the observed response at stage two. A path with no responses has weaker evidence. Thus the ordering tail consists of the paths “response at stage one” and “failure, then response.”
Under response probability \(p\), those paths have probabilities \(p\) and \((1-p)p\). The one-half calibration equation is \(p+(1-p)p=0.5\), or \(2p-p^2=0.5\). Solving gives \(p=1-\sqrt{0.5}=0.2929\) (the other quadratic root exceeds one and is not a probability).
Therefore \(\hat p_{\mathrm{SO}}=0.2929\), and \(\hat\theta_{\mathrm{SO}}=0.2929-0.20=0.0929\). The estimated treatment-arm response probability is 29.29%, corresponding to an estimated increase of 9.29 percentage points over the known comparator rate.
For comparison, the stopped-sample maximum likelihood estimate is \(1/2=0.50\), because one response was observed among two participants. That calculation does not account for the fact that the trial would have stopped earlier if the first participant had responded. The difference illustrates how the stopping rule can alter an estimate; it does not imply that the ordering estimate is universally preferable under every loss function or model.
Worked Example: A Response at the Third Stage
Now suppose the hypothetical rule is to stop at the first response or after at most three participants. The observed path is failure, failure, response. Again take \(p_C=0.20\). Calculate the stagewise ordering estimate.
For a candidate \(p_0<0.5\), the stagewise p-values for a first response at stages one, two, and three are, respectively, \(p_0\), \(1-(1-p_0)^2\), and \(1-(1-p_0)^3\). These increase with the stage in this range. Consequently, the outcomes at least as strongly ordered as the observed third-stage response are the first-stage response, the second-stage response, and the third-stage response. The all-failure path is not in the upper evidence tail.
The probability of at least one response by three participants is \(p+(1-p)p+(1-p)^2p=1-(1-p)^3\). Set this probability to one-half: \(1-(1-p)^3=0.5\), so \(p=1-\sqrt[3]{0.5}=0.2063\). This solution is below 0.5, which verifies the ordering used to form the tail.
The estimated response probability is \(0.2063\), giving a risk-difference estimate of \(\hat\theta_{\mathrm{SO}}=0.2063-0.20=0.00630\), or 0.630 percentage points. The stopped-sample maximum likelihood estimate is \(1/3=0.3333\). The two answers differ because the ordering estimate accounts for the probability of seeing a response at any of the permitted stopping stages, not just the observed number of responses divided by the observed sample size.
Worked Example: A First-Stage Response
Under the same three-participant stopping rule, suppose the trial stops after a response from the first participant. Find the ordering estimate and effect estimate when \(p_C=0.20\).
For \(0<p_0<1\), the stagewise p-value for a first-stage response is \(q_1(p_0)=p_0\). A response at a later stage has a larger upper-tail p-value: for example, with one response at stage two, \(q_2(p_0)=2p_0-p_0^2>p_0\). Outcomes with no responses have p-value one. Thus the ordering tail for the observed first-stage response contains only the first-stage-response path.
That path has probability \(p\) under response probability \(p\), so the calibration equation is \(p=0.5\). It follows that \(\hat p_{\mathrm{SO}}=0.5000\), and \(\hat\theta_{\mathrm{SO}}=0.5000-0.20=0.3000\). The treatment-arm response probability estimate is 50.00%, and the estimated risk difference is 30.00 percentage points.
This result depends on the defined ordering and on the path probabilities in this particular design. It is not obtained by interpreting the stagewise p-value as a probability that the parameter has a given value.
Justification, Diagnostics, and Alternatives
The calculation follows directly from the meaning of an ordering tail. Fix a candidate value \(p_0\), list all possible stopped paths, and mark those whose stagewise evidence is at least as strong as the observed evidence. Under \(p_0\), the probability of this marked set is the rank probability of the observed outcome or a more extreme one. Solving for the candidate value that makes this tail probability one-half centers the observed rank under the specified ordering.
State the response distribution, independence or other dependence assumptions, all permitted looks, and every stopping condition.
Choose and state a statistic or p-value at each possible stopping stage, including its direction and how ties are handled.
For a candidate parameter value, include every permitted stopped path at least as extreme as the observed path under the chosen ordering.
Compute the probability of that tail under the candidate value, using the design’s path probabilities, and solve the one-half equation. Check that the assumed ordering still holds at the solution.
In practice, the most important diagnostic is to verify the outcome space and its probabilities. For a small discrete design, enumerate the stopped paths and confirm that their probabilities sum to one. For larger designs, recursive calculations or simulation can be used, but simulation error should be reported or made sufficiently small for the intended precision. Also check that the solution lies in the parameter space and that changing the candidate value has not changed the ordering in a way the calculation ignored.
A stagewise ordering estimate is not automatically a maximum likelihood estimate, a bias-adjusted likelihood estimate, or an estimate with a guaranteed confidence-interval coverage property. The maximum likelihood estimate maximizes the likelihood for the observed data under its stated likelihood construction. A conditional likelihood estimate conditions on a specified event. The stagewise ordering estimate instead targets a one-half probability for a selected ordering tail. These procedures can differ because they answer different estimation questions.
| Approach | What it uses | Key distinction |
|---|---|---|
| Stopped-sample maximum likelihood estimate | The likelihood assigned to the observed data | Does not, by itself, guarantee adjustment for the ordering induced by stopping. |
| Conditional likelihood estimate | A likelihood conditional on a specified adaptation or selection event | Conditions on an event; the event and resulting estimand must be stated. |
| Stagewise ordering estimate | The probability of outcomes at least as extreme under the chosen stagewise ranking | Depends on the ordering statistic, the stopping rule, and the tie convention. |
There are important limitations. An ordering based on a one-sided statistic is suited to estimating in that direction; it does not automatically provide a symmetric treatment-effect summary when evidence in both directions matters. A different two-sided ordering must be specified for that purpose. Discrete outcomes can create ties or prevent an exact tail probability of one-half, and different tie conventions can yield different estimates. Misspecified response models, dependence between responses, or an incompletely represented stopping rule also invalidate the calculated path probabilities.
Common Mistake
Another frequent error is to use only the probability of the observed path in the calibration equation. The relevant probability is the probability of the entire ordering tail, which may include several paths that are at least as extreme as the observed one. Omitting those paths changes the equation and does not implement the defined ordering.
Check Your Understanding
Use the one-sided Bernoulli ordering and stopping rules defined above. Solutions are included after the questions.
- Under the two-participant rule, what is the ordering tail for an observed first-stage response?
- For the three-participant rule, write the one-half equation for an observed failure, failure, response path.
- Why is the probability of the observed path alone not the correct calibration probability for the second-stage response example?
- State one assumption or design detail that must be checked before applying this method to a real adaptive trial.
Solutions. (1) The tail contains only the first-stage-response path, so its probability under candidate \(p\) is \(p\). (2) The tail contains a response at any of the three stages, giving \(p+(1-p)p+(1-p)^2p=0.5\). (3) The ordering tail also includes paths with stronger evidence than the observed path; the calibration uses the probability of the whole tail. (4) For example, responses must follow the assumed Bernoulli model, and the stopping rule must be fully represented. If allocation, dependence, or other adaptation changes path probabilities, those features must be included in the calculation.