Why Plot a Spending Design?
A table of planned information times, spending targets, and efficacy boundaries can look plausible while still containing a transcription, indexing, or calibration error. Plotting these quantities in a common view makes their relationships easier to inspect: cumulative spending should progress toward the target type I error, increments show how that spending is allocated, and efficacy boundaries show the test-statistic thresholds used at each look.
Plots are diagnostic, not a substitute for calibration or error-control verification. A visually smooth curve does not establish that the joint probability of crossing a boundary equals the spending target. Conversely, a boundary that rises or falls unexpectedly is a reason to investigate, not automatically proof of an error: its shape depends on the spending function, information times, and the sequential model.
This discussion uses a one-sided efficacy test with upper boundaries and a canonical null model. Let \(t_k\) be the information fraction at look \(k\), with \(0<t_1<\cdots<t_K=1\). Under \(H_0\), the standardized statistics satisfy \(Z_k\sim N(0,1)\) and have correlation \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}\) for \(i<j\). The design uses fixed, correctly ordered information times and calibrated upper boundaries \(b_1,\ldots,b_K\). These assumptions give the joint null distribution needed to interpret cumulative crossing probabilities.
For a calibrated design, the recursion targets \(B_k=A(t_k)\). The increment \(\Delta A_k\) is the increase in the cumulative target, not generally the marginal probability \(P_0(Z_k\ge b_k)\). The latter includes paths that may already have crossed an earlier boundary. Confusing these quantities is a frequent source of misleading plots and implementation checks.
Constructing the Three Diagnostic Views
Use information fraction, not calendar time or look number, on the horizontal axis. The vertical axis for cumulative spending is probability, often displayed as a percentage. Plot \(A(t_k)\) at each planned look and connect the points only as a visual aid; the design targets are specified at those looks. A second view plots \(\Delta A_k\), which makes front-loaded or back-loaded allocation visible. A third view plots \(b_k\) against \(t_k\), preferably with the cumulative targets or actual crossing probabilities available for reference.
Basic checks follow directly from the definitions. Cumulative spending must be nondecreasing at the planned looks, and for a nonnegative-spending design each increment must be nonnegative. The final target must equal the design's total one-sided type I error, subject to the stated convention for the endpoint. The boundary plot should have one value per planned look, in the same order as the information-time vector. There is no universal rule that the boundary must be monotone for every spending function; compare its shape with the specified design and check it against the probability targets.
Evaluate \(A(t_k)\) at the actual planned information fractions. Include the target total \(\alpha\) as a reference at the final look.
Compute \(\Delta A_k=A(t_k)-A(t_{k-1})\), using the same ordering and endpoint convention. Check for negative values and verify that the increments sum to the final cumulative target.
Display \(b_k\) at each \(t_k\), then independently evaluate \(B_k=P_0(\max_{j\le k}(Z_j-b_j)\ge0)\). Compare \(B_k\) with \(A(t_k)\) at every look.
Worked Example: Reading a Cumulative Spending Curve
Consider a hypothetical one-sided design with \(\alpha=0.025\) and the O’Brien–Fleming spending function \(A(t)=1-\Phi\!\left(z_{1-\alpha}/\sqrt{t}\right)\), where \(z_{1-\alpha}\) is the standard normal quantile with lower-tail probability \(1-\alpha\). Here \(z_{0.975}=1.95996\). Suppose the planned information fractions are \(0.25, 0.50, 0.75,\) and \(1.00\).
At \(t=0.25\), the normal quantile in the upper tail is \(1.95996/\sqrt{0.25}=3.91992\), so \(A(0.25)=1-\Phi(3.91992)\approx0.0000444\). At \(t=0.50\), it is \(1.95996/\sqrt{0.50}=2.77180\), giving \(A(0.50)\approx0.00279\). At \(t=0.75\), it is \(1.95996/\sqrt{0.75}\approx2.263\), giving \(A(0.75)\approx0.0118\). At the endpoint, \(A(1)=1-\Phi(1.95996)=0.0250\).
The increments are approximately \(0.0000444\), \(0.00279-0.0000444=0.00275\), \(0.0118-0.00279=0.0090\), and \(0.0250-0.0118=0.0132\). Their sum is approximately \(0.0250\), apart from rounding. The plot therefore shows little planned spending early and larger increments later. That is consistent with the chosen spending function, but does not by itself establish that any accompanying boundaries are calibrated.
Boundary Plots Must Agree With Probability Checks
For a boundary sequence, the cumulative crossing probability at look \(k\) is a joint probability. It is not obtained by adding the marginal tail probabilities \(P_0(Z_j\ge b_j)\), because the statistics are correlated and crossing events overlap. Recursive calibration accounts for paths that have not crossed earlier. This is why a boundary plot should be paired with a numerical check of \(B_k\), using the joint null model and the same boundary values.
A simple necessary check is available without integrating the full joint distribution. Since crossing at look \(k\) is one of the events included in crossing by look \(k\), \(P_0(Z_k\ge b_k)\le B_k\). Thus, if the target is \(A(t_k)\), a calibrated boundary must satisfy \(P_0(Z_k\ge b_k)\le A(t_k)\), or equivalently \(b_k\ge z_{1-A(t_k)}\) for a one-sided standard normal statistic. This is only a necessary condition: satisfying it does not prove that earlier crossing paths have been handled correctly.
Worked Example: Spotting an Impossible Boundary Output
Suppose an analyst claims that a hypothetical three-look design with \(t=(0.25,0.50,1.00)\), the O’Brien–Fleming targets above, and a recursive canonical-null calibration produced \(b=(3.920,2.768,1.959)\), rounded to three decimals. Check the claimed second and final boundaries before trusting the plot.
At look 2, \(Z_2\sim N(0,1)\), so the marginal crossing probability at that look is \(P_0(Z_2\ge2.768)=1-\Phi(2.768)\approx0.00282\). But the claimed cumulative target is \(A(0.50)\approx0.0027873\). Since crossing by look 2 includes crossing at look 2, \(B_2\ge0.00282>0.0027873\). The boundary cannot achieve its claimed cumulative target.
At the final look, \(P_0(Z_3\ge1.959)=1-\Phi(1.959)\approx0.02506\), which exceeds \(A(1)\approx0.0250\). Therefore \(B_3\) must be at least \(0.02506\), also exceeding the target. These numbers are not a valid calibrated output. A plot can flag this problem, but the marginal-tail inequalities establish the contradiction directly; the boundaries must be higher than the displayed values to pass even these necessary checks.
Simulation and Plot Diagnostics
A simulation check can add the empirical cumulative crossing proportions \(\widehat B_k\) to the target plot. For each simulated null path, record whether any boundary has been crossed by each look; the cumulative indicators are nested across looks. With \(M\) simulated paths, the Monte Carlo standard error of an estimated crossing probability \(\widehat B_k\) is approximately \(\sqrt{\widehat B_k(1-\widehat B_k)/M}\). This quantifies simulation noise; it does not replace accurate integration or calibration.
Worked Example: Interpreting Simulation Discrepancies
Suppose a hypothetical simulation uses \(M=100{,}000\) null paths for the three-look O’Brien–Fleming design. It records empirical cumulative crossing proportions \(0.00005\), \(0.00290\), and \(0.02530\), against targets approximately \(0.0000444\), \(0.0027873\), and \(0.0250\).
At look 2, the discrepancy is \(0.00290-0.0027873=0.0001127\). The approximate Monte Carlo standard error is \(\sqrt{0.00290(1-0.00290)/100{,}000}\approx0.000170\). This difference is smaller than one standard error and is not, by itself, strong evidence of a problem. At the final look, the difference is \(0.02530-0.0250=0.00030\), while the standard error is \(\sqrt{0.02530(1-0.02530)/100{,}000}\approx0.000497\). This is also within one standard error.
The three cumulative estimates are based on the same simulated paths and are therefore dependent; they must not be treated as three independent tests. A sensible follow-up is to repeat the simulation with a larger \(M\), inspect the path-level crossing logic, and compare with a numerical joint-probability calculation. These hypothetical results are compatible with ordinary Monte Carlo variation, not proof that the code is correct.
What Plots Can and Cannot Diagnose
A useful diagnostic display makes errors visible without over-interpreting appearance. Check the following features:
- Ordering and alignment: every target, increment, and boundary must correspond to the same ordered information fractions. A shifted vector can produce a smooth but incorrect plot.
- Cumulative and incremental consistency: the plotted cumulative values should equal the running sum of the plotted increments, within rounding. Negative increments or a final sum that misses \(\alpha\) require explanation.
- Boundary plausibility: compare each boundary's marginal upper-tail probability with the cumulative target. A marginal tail larger than its target proves that the claimed calibration is impossible.
- Actual versus target spending: plot \(B_k\) from numerical integration or simulation alongside \(A(t_k)\). A visible discrepancy merits investigation, with simulation uncertainty considered when applicable.
- Information-time inputs: verify that the values are strictly increasing, on the intended scale, and consistent with the information fractions used in calibration.
These views have limits. Connecting points between looks does not define spending at unplanned times unless the design specifies how to evaluate \(A(t)\) there. Rounded values can obscure small discrepancies, particularly at early looks where spending is tiny. A log scale may help inspect small increments, but the numerical values should remain available. Finally, an attractive boundary curve cannot reveal every error in the joint probability calculation; independent verification remains necessary.
| Diagnostic view | Main question | What it cannot establish alone |
|---|---|---|
| Cumulative \(A(t_k)\) | Does planned spending rise coherently to the total? | That the calibrated boundaries achieve those targets |
| Incremental \(\Delta A_k\) | How is the target allocated across looks? | That increments equal marginal boundary-crossing probabilities |
| Boundary \(b_k\) | Are values aligned with information times and plausible? | That cumulative type I error is controlled |
| Actual \(B_k\) versus target | Does joint crossing probability match spending? | That simulation error is negligible or code is otherwise correct |
Common Mistake and Practical Interpretation
A common mistake is to label \(\Delta A_k\) as “the probability of crossing at look \(k\)” or to equate it with \(P_0(Z_k\ge b_k)\). The increment is the increase in the target cumulative error probability. The actual probability of first crossing at look \(k\) is \(B_k-B_{k-1}\), which equals \(\Delta A_k\) when calibration achieves both cumulative targets exactly. The marginal tail at look \(k\) is generally larger because it counts paths that may already have crossed. Keep these three quantities distinct in tables, code, and plot labels.
In practice, use plots to locate anomalies, then verify them numerically. The strongest checks are algebraic consistency of the spending values, the marginal-tail lower bound on cumulative crossing, and direct evaluation of the joint crossing probabilities. This combination distinguishes a harmless visual feature or Monte Carlo fluctuation from a genuine design inconsistency.
Check Your Understanding
Answer each question, then compare with the solutions.
- If cumulative spending at two successive looks is \(0.004\) and \(0.010\), what is the second look's incremental allocation?
- Why is \(\Delta A_k\) not generally equal to \(P_0(Z_k\ge b_k)\)?
- A claimed target is \(A(t_k)=0.006\), while the marginal tail at the reported boundary is \(P_0(Z_k\ge b_k)=0.007\). What can be concluded?
- With \(M=50{,}000\) simulations and empirical cumulative crossing proportion \(\widehat B_k=0.02\), calculate the approximate Monte Carlo standard error.
- Does a smooth, increasing cumulative spending plot prove that the final group sequential design controls type I error? Explain.
Solutions. (1) The increment is \(0.010-0.004=0.006\). (2) \(\Delta A_k\) is a difference between cumulative targets; the marginal event \(Z_k\ge b_k\) includes paths that may have crossed earlier, whereas an incremental first-crossing probability excludes those paths. (3) Since \(B_k\ge P_0(Z_k\ge b_k)=0.007\), the cumulative probability must exceed the claimed target of \(0.006\); the reported boundary cannot be calibrated to that target. (4) The standard error is approximately \(\sqrt{0.02(0.98)/50{,}000}=\sqrt{0.000000392}\approx0.000626\). (5) No. The plot checks the target's shape, but control depends on the joint null crossing probabilities under the model and the calibrated boundaries; those probabilities must be verified separately.