Why Calibration Needs More Than Unequal Limits
An asymmetric design can assign different error budgets to its upper and lower directions, use different boundary shapes, or both. Unequal limits alone do not establish that the intended error budgets have been achieved. Calibration is a property of the complete stopping rule under the null: it depends on the sequence of boundaries, the null distribution of the interim statistics, their dependence across looks, and the direction assigned to each crossing.
The practical diagnostic is therefore threefold. First, compute the probability of a first crossing in each direction under the null and compare it with its own target. Second, confirm that the statistic and boundaries use the intended sign and null reference value. Third, trace representative paths to verify that the rule stops at the first boundary reached and reports the correct direction. A design can pass one check and fail another; for example, its numerical error rates may be correct in a program that labels benefit and harm backwards.
For calibration, state the null model before calculating either probability. In the canonical Gaussian model under the null value \(\theta_0\), the statistics have mean zero, unit variance, and correlation \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}\) for information times \(t_i\le t_j\). This is the standardized Brownian-motion model; it assumes the specified information times and the canonical joint distribution are appropriate. Other valid designs may use a different joint null model, but the directional error calculation must use the model that matches the design.
Write \(\alpha_U\) for the probability of first stopping at an upper boundary under \(H_0\), and \(\alpha_L\) for the corresponding lower-boundary probability. The two first-stop events are disjoint, so the overall two-sided null rejection probability is \(\alpha_U+\alpha_L\). If the design specifies separate directional targets, each probability must be checked against its own target; matching only the sum is not sufficient.
What the Calibration Check Calculates
Let \(C_{k-1}\) denote the event that the trial has not crossed either boundary at any earlier look. The directional probabilities are sums of first-passage probabilities:
The continuation event matters. Adding the marginal probabilities \(P_{H_0}(Z_k\ge U_k)\) across looks would count paths that crossed earlier, and adding marginal upper and lower tails does not account for the joint stopping rule. The earlier material on joint null error across two tails establishes why the total error depends on both tails and the dependence across looks. For an audit, the useful quantities are the first-passage probabilities above.
For two looks with \(t_1/t_2=1/2\), the canonical model gives \(Z_2\mid Z_1=z\sim N(z/\sqrt{2},1/2)\). If the first-look continuation interval is \(L_1<Z_1<U_1\), the upper error probability is
Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The first term is an upper stop at look 1. The integral counts only paths that continue after look 1 and then cross the upper limit at look 2. The lower error has the analogous form, using the conditional probability of \(Z_2\le L_2\). In practice, these integrals or their multi-look counterparts are evaluated using numerical integration or a validated group-sequential calculation.
Worked Example: Checking Directional Targets at One Look
Suppose a hypothetical one-look design uses \(Z\sim N(0,1)\) under \(H_0\), with upper limit \(U=2.05\) and lower limit \(L=-2.33\). Its target directional error rates are \(\alpha_U=0.020\) and \(\alpha_L=0.010\). Check each target and the overall null rejection probability.
At one look, there is no earlier continuation condition. The upper directional error is \(P(Z\ge2.05)=1-\Phi(2.05)\approx0.02018\). The lower directional error is \(P(Z\le-2.33)=\Phi(-2.33)\approx0.009903\).
The upper probability exceeds its target by approximately \(0.02018-0.02000=0.00018\), while the lower probability is below its target by approximately \(0.01000-0.009903=0.000097\). These differences are small, consistent with rounding the limits to two decimal places, but the limits do not exactly attain both targets. The overall rejection probability is \(0.02018+0.009903=0.03008\), because the one-look upper and lower events are disjoint.
This check is valid under the stated standard normal null model and the stated sign convention. It does not establish calibration for a multi-look design: additional looks change first-passage probabilities, even if the marginal distribution at each look is standard normal.
Check the Sign Convention and the Actual Stopping Rule
Before evaluating a probability, write down what a positive statistic means in the application. If larger values of the effect estimate represent benefit, the upper boundary may be the benefit stop and the lower boundary the harm stop. If the statistic is instead defined with the opposite contrast, those interpretations reverse. The labels must follow the defined statistic, not the visual position of a boundary in a table or software output.
Also verify the reference value used in standardization. For a null value \(\theta_0\), a typical statistic has the form \(Z_k=(\hat{\theta}_k-\theta_0)/SE_k\), with any required design-specific scaling. Replacing \(\theta_0\) by zero when the null is nonzero shifts the statistic and changes the operating characteristics. A correct set of limits applied to the wrong standardized statistic is not the calibrated procedure.
For path diagnostics, examine paths that approach each boundary, paths that cross a boundary early, and paths that would cross the opposite boundary at a later look. The rule is based on the first crossing. Once the trial stops, later hypothetical values do not change the recorded decision. These checks are implementation diagnostics: they test whether the programmed rule agrees with the design specification, not whether a simulated path proves calibration.
Worked Example: Auditing Two-Look Decisions on Representative Paths
Consider a hypothetical two-look design with \(L_1=-2.4,\ U_1=2.6,\ L_2=-2.3,\ U_2=2.1\). At each look, stop for an upper crossing if \(Z_k\ge U_k\), for a lower crossing if \(Z_k\le L_k\), and continue otherwise. Classify the following paths.
Path A: \(Z_1=2.30,\ Z_2=2.15\). At look 1, \(-2.4<2.30<2.6\), so the trial continues. At look 2, \(2.15\ge2.1\), so the first crossing is upper. The decision is an upper stop at look 2.
Path B: \(Z_1=-2.45,\ Z_2=2.8\). At look 1, \(-2.45\le-2.4\), so the trial stops at the lower boundary. The later value \(2.8\) is not used; reporting an upper stop would violate the first-crossing rule.
Path C: \(Z_1=1.0,\ Z_2=-2.35\). The first value lies between \(-2.4\) and \(2.6\), so the trial continues. Since \(-2.35\le-2.3\), the trial stops at the lower boundary at look 2.
Path D: \(Z_1=2.7,\ Z_2=-3.0\). The first value already satisfies \(2.7\ge2.6\), so the trial stops at the upper boundary at look 1. The later lower value cannot turn that decision into a lower stop. These examples check the inequalities, look order, and directional labels; they do not by themselves determine the null error probabilities.
Worked Example: Diagnosing a Shifted Null and an Effect-Scale Threshold
Suppose, hypothetically, that positive values of an effect \(\theta\) indicate benefit, the null value is \(\theta_0=0.40\) effect units, and the standard error at an interim look is \(SE=0.15\) effect units. The standardized statistic is \(Z=(\hat{\theta}-0.40)/0.15\), and the design uses \(U=2.2\) and \(L=-2.8\). Find the effect estimates corresponding to the two boundaries and identify a common implementation error.
Solving \(Z=U\) for the estimate gives \(\hat{\theta}=0.40+2.2(0.15)=0.730\) effect units. Solving \(Z=L\) gives \(\hat{\theta}=0.40-2.8(0.15)=-0.020\) effect units. Thus, at this look, the upper stop occurs when the estimate reaches at least \(0.730\), while the lower stop occurs when it reaches at most \(-0.020\), under the stated standardization.
If an implementation mistakenly uses \(Z=\hat{\theta}/0.15\), it has silently centered the statistic at zero rather than at the null value \(0.40\). It would place the corresponding estimate thresholds at \(0.330\) and \(-0.420\) effect units. Those are different stopping decisions. The correct diagnostic is to check the numerator against the specified null value and confirm that larger \(Z\) still represents the intended upper direction. The calculation assumes the supplied standard error and null reference are appropriate for this hypothetical look.
Calibration Diagnostics and Failure Modes
A useful audit keeps the specification, probability calculation, and path implementation separate. This makes it easier to locate a failure rather than merely noticing that a final total looks plausible.
Write \(\alpha_U\), \(\alpha_L\), the null value, the definition of \(Z_k\), the information times, and the assumed joint null distribution. Confirm that the directional targets are not being confused with their sum.
At every look, verify \(L_k<U_k\), the exact crossing inequalities, and the mapping of upper and lower to benefit or harm. Check any shifted coordinate against the specified null value.
Use the full joint null model and count only paths that have not crossed earlier. Report \(\alpha_U\) and \(\alpha_L\) separately, as well as their sum.
Test values just inside and outside each limit, paths that cross at the first look, and paths whose later values cross the opposite boundary. Confirm that the recorded decision is the first crossing.
Compare each calculated directional probability with its own target, allowing only the documented numerical tolerance. If a discrepancy is too large, inspect the null centering, signs, boundaries, information times, dependence model, and stopping code.
A common calibration failure is to verify only that \(\alpha_U+\alpha_L\) is close to the overall target. For example, a design with targets \(0.02\) and \(0.01\) could produce \(0.025\) and \(0.005\): the total remains \(0.030\), but neither directional allocation is met. Another failure is summing marginal tail probabilities across looks. That calculation ignores prior stopping and cross-look dependence; it is not the first-passage error probability.
Other warning signs include a boundary that is labelled “harm” even though the statistic's positive direction represents harm, a reported decision that changes when later values are supplied after stopping, or discrepancies between a probability calculation and a path test at a boundary equality. The specification must settle whether equality counts as crossing; the convention here uses \(Z_k\ge U_k\) and \(Z_k\le L_k\). In a continuous Gaussian model, equality has probability zero, but exact equality can occur in rounded or discrete software inputs, so the implementation should still be explicit.
Choosing a Diagnostic Approach and Knowing Its Limits
For a small number of looks under a canonical Gaussian model, numerical integration of the first-passage probabilities is direct and helps expose the role of dependence and continuation regions. A validated group-sequential calculation can handle more looks and standard boundary families efficiently. Simulation is a useful independent check: generate paths under the null model, apply the programmed first-crossing rule, and estimate the proportion stopping in each direction. Its estimates have Monte Carlo uncertainty, so simulation alone may not resolve small deviations from a target without a sufficiently large number of paths.
These approaches answer related but distinct questions. Analytic or numerical calculation evaluates the specified model; simulation can check the implementation under that model; representative paths check decision logic. None substitutes for the others. A calculation using the wrong null distribution can be precise but irrelevant, while a few correctly classified paths cannot establish error-rate calibration.
Calibration is conditional on the model, statistic, information schedule, boundary values, and stopping rule being audited. Deviations from the assumed information structure or null distribution can alter operating characteristics. Rounding limits may also cause small deviations, and an audit should distinguish those from substantive errors using a prespecified tolerance. Finally, calibration under the null does not by itself establish adequate power or desirable expected stopping behavior under alternatives; those are different design properties.
Check Your Understanding
Answer each question using the definitions and checks in this tutorial.
- A two-look design has directional targets \(0.025\) upper and \(0.015\) lower. A calculation gives \(0.030\) and \(0.010\). Is the overall target sum alone sufficient to declare the directional calibration successful?
- Why is adding \(P(Z_1\ge U_1)\) and \(P(Z_2\ge U_2)\) generally not the upper first-passage error probability?
- At a first look, the lower limit is \(-2.0\). A path has \(Z_1=-2.1\) and a later hypothetical value \(Z_2=3.0\). What is the stopping decision?
- If \(Z=(\hat{\theta}-0.5)/0.2\), what estimate corresponds to an upper boundary \(U=2.0\), and what null-centering error should an auditor look for?
Solutions. (1) No. Both directional rates must be compared separately with their targets. The total is \(0.030+0.010=0.040\), matching the target sum \(0.025+0.015=0.040\), but the upper rate is \(0.005\) too high and the lower rate \(0.005\) too low. (2) The second marginal tail includes paths that may already have stopped at look 1, and the calculation omits the cross-look dependence and continuation condition. (3) Stop at the lower boundary at look 1; the later value cannot change the first crossing. (4) Rearranging gives \(\hat{\theta}=0.5+2.0(0.2)=0.90\) effect units. Check that the implementation subtracts the null value \(0.5\), rather than zero, and that larger standardized values represent the intended upper direction.