Likelihood-Ratio Ordering for Stopped Outcomes

Why Order Stopped Outcomes by a Likelihood Ratio?

After an interim look, the observed data include not only a test statistic but also the fact that the trial stopped at that look. A p-value therefore needs an ordering of outcomes from the stopped experiment, followed by a null probability calculation using that experiment’s stopping rule. Likelihood-ratio ordering provides one way to rank those outcomes: it compares how well the observed data fit the null hypothesis with how well they fit a specified alternative.

The alternative matters. An outcome is considered more compelling against the null when it is relatively more likely under the alternative than under the null. For a fixed, prespecified alternative, this comparison gives a precise evidence scale. It does not, by itself, make every possible likelihood-ratio statistic a valid p-value; calibration still requires the null distribution of the stopped outcome.

Definition: Let \(X_{\mathrm{stop}}\) denote all data observed up to and including the stopping look, and let \(f_0(x)\) and \(f_1(x)\) be its probability mass functions or densities under \(H_0\) and a specified alternative \(H_1\). The null-to-alternative likelihood ratio is \(R(x)=f_0(x)/f_1(x)\). Smaller values indicate outcomes relatively more likely under \(H_1\). For an observed outcome \(x_{\mathrm{obs}}\), the likelihood-ratio ordered p-value is \(p_{\mathrm{LR}}=P_0\{R(X_{\mathrm{stop}})\leq R(x_{\mathrm{obs}})\}\).

The likelihoods must refer to the data as observed under the stopping plan. In a continuous model, densities replace probabilities. If different outcomes have the same ratio, the nonrandomized definition includes the entire tie, which can make the p-value conservative. Here the examples use continuous outcomes, so ties have probability zero.

Likelihood Ratios for Normal Group Sequential Data

Consider a one-sided normal model with a known variance and independent increments. At look \(k\), let \(S_k\) be the cumulative score and \(I_k\) the cumulative information. The increment from the preceding look has distribution \(\Delta S_j\sim N(\theta\Delta I_j,\Delta I_j)\), independently across looks. Test \(H_0:\theta=0\) against the specified simple alternative \(H_1:\theta=\theta_1\), where \(\theta_1>0\). The information schedule, the stopping boundaries, and the rule for ending the trial are treated as fixed in advance.

For data observed through look \(k\), the log likelihood under \(\theta\), apart from terms not depending on \(\theta\), is \(\theta S_k-\theta^2 I_k/2\). Thus the likelihood ratio and the evidence scale are

R_k=\frac{f_0}{f_{\theta_1}} =\exp\left(-\theta_1 S_k+\frac{\theta_1^2 I_k}{2}\right), \qquad E_k=\log\left(\frac{f_{\theta_1}}{f_0}\right) =\theta_1 S_k-\frac{\theta_1^2 I_k}{2}.

The likelihood-ratio ordering ranks larger \(E_k\) (equivalently, smaller \(R_k\)) as more supportive of the specified alternative. Crucially, \(E_k\) depends on both the score and the information at the stopping look. Comparing standardized \(Z\)-statistics alone need not give this ordering.

Why the Stopping Rule Belongs in the Calculation

Suppose the complete observed path through look \(k\) is \(x=(x_1,\ldots,x_k)\). A deterministic stopping rule assigns a stopping look to that path. The density of the stopped path can be written as the ordinary joint density through \(k\), multiplied by an indicator that the rule stops at \(k\) for that path. On paths that can occur under the rule, this indicator is the same under \(H_0\) and \(H_1\). It therefore cancels in the likelihood ratio. The path still must satisfy the stopping rule, however, when calculating the null probability of outcomes at least as extreme as the observation.

For a continuous stopped outcome, the ordered p-value is valid under the simple null because it is the null probability of the lower tail of the null-distributed ratio \(R(X_{\mathrm{stop}})\). In particular, for any \(u\) between zero and one, \(P_0(p_{\mathrm{LR}}\leq u)\leq u\); equality holds when the ratio has a continuous null distribution. The inequality allows for ties or other discreteness. This is the superuniformity criterion for a valid p-value.

Worked Example: A Two-Look Normal Likelihood-Ratio P-Value

Hypothetical design. Let the information at the first look be \(I_1=1\), with final information \(I_2=2\). Under \(H_0\), \(S_1\sim N(0,1)\) and the independent increment \(S_2-S_1\sim N(0,1)\). Stop at the first look if \(S_1\geq2\); otherwise continue to the final look. Test \(H_0:\theta=0\) against the specified alternative \(\theta_1=1\). Suppose the observed first-look score is \(S_1=2.4\), so the trial stops early.

At this observation, \(E_1=1(2.4)-1^2(1)/2=1.9\), and the null-to-alternative ratio is \(R_{\mathrm{obs}}=\exp(-1.9)\approx0.1496\). An early-stop outcome has \(R_1\leq R_{\mathrm{obs}}\) exactly when \(S_1\geq2.4\). A final-look outcome has \(R_2\leq R_{\mathrm{obs}}\) exactly when \(-S_2+2/2\leq-1.9\), or \(S_2\geq2.9\). Final-look outcomes are possible only when \(S_1<2\).

Consequently the null probability of the ordered tail is

\begin{aligned} p_{\mathrm{LR}} &=P_0(S_1\geq2.4)+P_0(S_1<2,\ S_2\geq2.9)\\ &=\{1-\Phi(2.4)\} +\int_{-\infty}^{2}\phi(x)\{1-\Phi(2.9-x)\}\,dx\\ &\approx 0.00820+0.01320=0.0214. \end{aligned}

Here \(\phi\) and \(\Phi\) are the standard normal density and distribution function. The integral conditions on \(S_1=x\); the remaining increment is independent and standard normal. Equivalently, the second term is \(P_0(S_2\geq2.9)-P_0(S_1\geq2,S_2\geq2.9)\), which gives the same numerical calculation. The resulting likelihood-ratio ordered p-value is about 0.0214. It is a null probability for the specified stopping plan and alternative, not the probability that \(H_0\) is true.

Reading the Ordering Across Looks

The same ratio formula compares outcomes from different stopping looks, but the information term changes the score threshold. For a fixed observed evidence value \(c\), an outcome at look \(k\) is at least as extreme when \(\theta_1 S_k-\theta_1^2 I_k/2\geq c\). Thus, for \(\theta_1>0\), its score must satisfy \(S_k\geq(c+\theta_1^2 I_k/2)/\theta_1\). The ordered null tail is the probability of meeting this evidence condition at whichever look the design actually stops.

Worked Example: Comparing an Early and a Final Outcome

Use the hypothetical design above. Compare the observed early-stop outcome \(S_1=2.4\) with a possible final outcome having \(S_1=1.5\) and \(S_2=2.7\). The latter is a valid final outcome because \(S_1<2\). Under \(\theta_1=1\), the early outcome has \(E_1=2.4-0.5=1.9\), so \(R_1=\exp(-1.9)\approx0.1496\). The final outcome has \(E_2=2.7-2/2=1.7\), so \(R_2=\exp(-1.7)\approx0.1827\).

Since \(0.1496<0.1827\), the early-stop outcome is ranked as more supportive of \(\theta_1=1\), even though both outcomes have positive scores. The conclusion follows from the full likelihood ratio, not merely from comparing \(S_1\) and \(S_2\): the final outcome’s likelihood incorporates the additional information available at look 2.

Worked Example: The Specified Alternative Changes the Ranking

Keep the same two hypothetical paths, but now compare them under the specified alternative \(\theta_1=0.5\). For the early outcome, \(E_1=0.5(2.4)-0.5^2(1)/2=1.075\), giving \(R_1=\exp(-1.075)\approx0.3413\). For the final outcome, \(E_2=0.5(2.7)-0.5^2(2)/2=1.10\), giving \(R_2=\exp(-1.10)\approx0.3329\).

The final outcome now has the smaller ratio and is ranked as more supportive of this alternative. The data and stopping rule did not change; the alternative did. Likelihood-ratio ordering is therefore not an alternative-free ranking of stopped outcomes. The alternative must be selected for a scientifically meaningful reason and specified before interpreting the resulting p-value.

Assumptions, Diagnostics, and Alternatives

The normal-increment calculation relies on independent increments with the stated means and variances, known information values, a prespecified alternative, and a correctly implemented stopping rule. In a trial, the assumed model should match the analysis scale and information schedule. If the increments are not independent or their null distribution is misspecified, the calculated ordered tail can be wrong. The stopping rule must also be represented accurately: omitting an efficacy or futility boundary changes which paths can appear at each look and therefore changes the null probability.

A useful diagnostic is to enumerate the possible stopping regions and check that the calculation includes each one exactly once. For a two-look design, the first-look region and the continuation region partition the null outcomes. In the worked calculation, early outcomes contribute only when \(S_1\geq2.4\), while final outcomes contribute only when \(S_1<2\) and \(S_2\geq2.9\). Including final paths with \(S_1\geq2\) would incorrectly count paths that the design would already have stopped.

Likelihood-ratio ordering differs from stagewise ordering, which ranks outcomes using the stagewise evidence convention. The two orderings need not give the same p-value because they can rank stopped outcomes differently. Conditional p-values given the stopping look address a different reference distribution: they condition on the observed look, whereas the likelihood-ratio ordered p-value here uses the null distribution across all stopping looks. These approaches answer different questions, and the intended ordering and reference distribution should be stated.

The simple-null calculation also has limits. A likelihood ratio formed for one fixed alternative does not automatically provide a valid p-value for a composite null or for an alternative selected after inspecting the data. For a composite null, calibration must protect against every null parameter value, for example by taking the largest relevant null tail probability over that parameter space. Nor does a small ordered p-value measure the size of the treatment effect or establish that the model is adequate.

Common Mistakes

Reversing the ratio. With \(R=f_0/f_1\), smaller values indicate greater relative support for \(H_1\). The ordered tail is \(R\leq R_{\mathrm{obs}}\), not \(R\geq R_{\mathrm{obs}}\).
Using only the observed statistic’s ordinary tail. The p-value must sum or integrate over all stopped outcomes at least as extreme under the chosen ordering, respecting which outcomes can occur at each look.
Ignoring information at the stopping look. The normal-model evidence is \(\theta_1S_k-\theta_1^2I_k/2\); score values from different looks cannot be compared by simply using the same raw-score cutoff.
Changing the alternative after seeing the data. That changes the ordering itself and invalidates the stated calibration unless the selection procedure is accounted for.

Key takeaway: Likelihood-ratio ordering ranks stopped paths by \(f_0/f_1\), with smaller ratios treated as more extreme against the null. A valid p-value is the probability, under the null and the actual stopping design, of obtaining an outcome at least as extreme under that prespecified ordering.

Check Your Understanding

Use the definitions and normal two-look model above. The questions are followed by brief solutions.

If \(R(x_{\mathrm{obs}})=0.20\), which likelihood-ratio tail defines the ordered p-value?
In the normal model with \(\theta_1=1\), \(S_2=3\), and \(I_2=2\), calculate \(E_2\) and \(R_2\).
Why must a final-look outcome in the two-look example also satisfy \(S_1<2\)?
Does a likelihood-ratio ordering necessarily rank outcomes the same way for every specified alternative?
What key information is lost if the p-value calculation uses the ordinary final-look normal tail without representing early stopping?

Solutions. (1) Use \(P_0\{R(X_{\mathrm{stop}})\leq0.20\}\), because smaller null-to-alternative ratios are more extreme. (2) \(E_2=3-2/2=2\), so \(R_2=\exp(-2)\approx0.1353\). (3) The design stops at look 1 whenever \(S_1\geq2\), so such a path cannot be a final-look outcome. (4) No. As the worked comparison shows, changing \(\theta_1\) can reverse the ranking. (5) It ignores the stopping regions and the distribution of outcomes across looks, so it does not generally give the ordered p-value for the stopped experiment.