Sensitivity Of Asymmetric Rules To Effect Direction

Why Effect Direction Matters When Boundaries Stay Fixed

An asymmetric sequential rule can treat positive and negative evidence differently by design. For example, a study may use a less stringent boundary for evidence of harm than for evidence of benefit, or may allocate different error budgets to the two directions. Once the boundaries have been prespecified, changing the assumed effect direction does not change the rule. It changes the distribution of the statistics evaluated against that rule, and therefore changes the operating interpretation.

The key distinction is between the rule and the effect scenario. The rule consists of the planned information levels and the upper and lower boundaries at each look. An effect scenario determines the mean drift of the sequential statistics. Sensitivity analysis asks what the same fixed rule does under modestly different positive and negative drifts; it does not silently redesign or recalibrate the boundaries.

Definition: At look \(k\), let \(I_k\) be the cumulative information and let \(Z_k\) be the standardized statistic, oriented so positive values support benefit and negative values support harm. A fixed asymmetric rule has prespecified upper and lower boundaries \(u_k\) and \(l_k\), with \(l_k<u_k\). It stops for benefit if \(Z_k\ge u_k\), stops for harm if \(Z_k\le l_k\), and otherwise continues, except that a terminal look must assign a final decision according to the prespecified rule.

Model and Directional Drift

For the calculations below, use the canonical normal approximation for a sequence of group sequential statistics. The schedule \(I_1<\cdots<I_K\) is fixed, the boundary values are fixed before examining the data, and the statistic has unit variance at each look. Under a constant effect \(\theta\) per square root of information, its mean at look \(k\) is \(\theta\sqrt{I_k}\). Thus

Z_k\sim N(\theta\sqrt{I_k},1), \qquad \operatorname{Corr}(Z_j,Z_k)=\sqrt{\frac{I_j}{I_k}}\quad (j<k).

This canonical model is a standard large-sample approximation for many sequential tests; it assumes the information measure and effect scale are correctly specified and that the planned information increments support the normal independent-increments model. In an actual design, nuisance-parameter estimation, nonnormal outcomes, or an incorrectly specified information schedule can affect the approximation. The examples use the stated model and treat \(\theta\) as the assumed effect for operating-characteristic calculations, not as an observed estimate.

At a particular look, the marginal crossing probabilities follow directly from the normal model:

P_\theta(Z_k\ge u_k) =1-\Phi\!\left(u_k-\theta\sqrt{I_k}\right), \qquad P_\theta(Z_k\le l_k) =\Phi\!\left(l_k-\theta\sqrt{I_k}\right).

Here \(\Phi\) is the standard normal cumulative distribution function. A positive change in \(\theta\) shifts the distribution to the right: the upper-tail crossing probability increases and the lower-tail crossing probability decreases. A negative change shifts it to the left. These statements concern crossing at that look; they do not by themselves give the probability of first stopping there or the probability of stopping in a given direction at any time.

Why Sign Reversal Is Not Symmetry

If the effect changes from \(\theta\) to \(-\theta\), then the distribution of \(-Z_k\) under \(\theta\) has the same mean and variance as the distribution of \(Z_k\) under \(-\theta\). Consequently,

P_{-\theta}(Z_k\ge u_k)=P_{\theta}(Z_k\le -u_k).

This identity compares the upper boundary \(u_k\) with the reflected value \(-u_k\). It does not say that the upper crossing probability under one direction equals the lower crossing probability under the opposite direction: that would require the lower boundary to be \(l_k=-u_k\). With asymmetric boundaries, the two probabilities generally differ. The same issue applies to whole sequential paths: sign reversal reflects every boundary, and only a sign-symmetric rule has the corresponding directional symmetry.

Key distinction: Changing the assumed effect direction changes the distribution of \(Z_k\), not the prespecified values \(u_k\) and \(l_k\). A mirrored probability comparison is valid only when the compared boundaries are actually mirror images, or when the reflected boundary is explicitly used.

Worked Look-by-Look Sensitivity

Worked Example: Opposite Effects at a Fixed Look

Suppose a hypothetical trial has a look at information \(I=64\), with fixed upper boundary \(u=2.4\) and lower boundary \(l=-1.8\). Compare the marginal crossing probabilities for assumed effects \(\theta=0.05\) and \(\theta=-0.05\), in the stated effect-per-square-root-information units.

The mean of \(Z\) is \(\theta\sqrt{I}\), and \(\sqrt{64}=8\). For \(\theta=0.05\), the mean is \(0.05(8)=0.40\). Therefore,

P(Z\ge 2.4)=1-\Phi(2.4-0.40) =1-\Phi(2.00)=0.0228,

and

P(Z\le -1.8)=\Phi(-1.8-0.40) =\Phi(-2.20)=0.0139.

For \(\theta=-0.05\), the mean is \(-0.40\). The corresponding probabilities are

P(Z\ge 2.4)=1-\Phi(2.4+0.40) =1-\Phi(2.80)=0.00256,

and

P(Z\le -1.8)=\Phi(-1.8+0.40) =\Phi(-1.40)=0.0808.

The fixed rule is more likely to cross its upper boundary under the positive assumed effect and its lower boundary under the negative assumed effect. The changes are not mirror images: the lower boundary is closer to zero than the upper boundary, so the negative scenario has a comparatively larger lower-tail crossing probability. These are probabilities of being beyond each boundary at this look, not an overall sequential stopping probability.

Effect-Scale Interpretation of Fixed Limits

The standardized distance between the mean at look \(k\) and a boundary can also be expressed as a threshold on the effect scale. The mean reaches the upper boundary when \(\theta\sqrt{I_k}=u_k\), and reaches the lower boundary when \(\theta\sqrt{I_k}=l_k\). Thus the corresponding effect-scale thresholds are \(u_k/\sqrt{I_k}\) and \(l_k/\sqrt{I_k}\). These are locations of the assumed mean, not deterministic cutoffs: sampling variability remains, so crossing probabilities are smooth rather than jumping from zero to one at these thresholds.

Worked Example: Comparing Directional Thresholds

At a hypothetical terminal look, suppose \(I=100\), \(u=2.0\), and \(l=-3.0\). The mean reaches the upper boundary at effect \(2.0/\sqrt{100}=0.20\), and reaches the lower boundary at effect \(-3.0/\sqrt{100}=-0.30\). Thus the fixed rule places the boundary locations at \(0.20\) and \(-0.30\) on the effect scale; it is not symmetric around zero.

Now compare assumed effects \(\theta=0.18\) and \(\theta=-0.18\). For \(0.18\), the statistic has mean \(0.18(10)=1.8\), so

P(Z\ge 2.0)=1-\Phi(2.0-1.8) =1-\Phi(0.20)=0.4207,

while the lower-boundary probability is

P(Z\le -3.0)=\Phi(-3.0-1.8) =\Phi(-4.8)\approx 0.000001.

For \(\theta=-0.18\), the mean is \(-1.8\). Then

P(Z\ge 2.0)=1-\Phi(2.0+1.8) =1-\Phi(3.8)\approx 0.000072,

and

P(Z\le -3.0)=\Phi(-3.0+1.8) =\Phi(-1.2)=0.1151.

Although the effects have equal magnitude and opposite signs, the probabilities of crossing in the corresponding directions differ substantially. This is consistent with the boundary locations: the assumed negative effect is farther from the lower boundary than the positive effect is from the upper boundary. Again, these are marginal probabilities at the terminal look under the normal model.

Multiple Looks: What a Stage Probability Does and Does Not Say

With several looks, each \(Z_k\) has the marginal distribution specified above, but the statistics are correlated. A statistic that crossed a boundary earlier would not ordinarily be observed as a continuing participant or study at a later look: the rule has already stopped it. Therefore, adding marginal boundary-crossing probabilities across looks double-counts some paths and does not yield the probability of stopping.

A first-stopping probability at look \(k\) requires the path to remain within the continuation region at every earlier look, then cross the relevant boundary at \(k\). In notation, for an upper stop at \(k\), the event is

\{l_j<Z_j<u_j\text{ for every }j<k,\; Z_k\ge u_k\}.

Its probability depends on the joint distribution across looks. The earlier material on expected stopping and two-sided alternatives distinguishes stagewise quantities from whole-path operating characteristics; detailed numerical integration of those paths is a separate calculation. For direction sensitivity, report clearly whether a number is a single-look marginal probability, a first-stopping probability, or a cumulative probability of ever crossing.

Worked Example: Sensitivity at Two Planned Looks

Consider hypothetical looks at \(I_1=25\) and \(I_2=100\), with fixed boundaries \((u_1,l_1)=(2.2,-2.8)\) and \((u_2,l_2)=(2.0,-2.5)\). Compare the marginal probabilities at each look for \(\theta=0.04\) and \(\theta=-0.04\). The schedule and boundaries are held fixed.

For \(\theta=0.04\), the means are \(0.04\sqrt{25}=0.20\) and \(0.04\sqrt{100}=0.40\). At look 1,

P(Z_1\ge2.2)=1-\Phi(2.0)=0.0228,\qquad P(Z_1\le-2.8)=\Phi(-3.0)=0.00135.

At look 2, the marginal probabilities are

P(Z_2\ge2.0)=1-\Phi(1.6)=0.0548,\qquad P(Z_2\le-2.5)=\Phi(-2.9)=0.00187.

For \(\theta=-0.04\), the means are \(-0.20\) and \(-0.40\). Thus at look 1,

P(Z_1\ge2.2)=1-\Phi(2.4)=0.00820,\qquad P(Z_1\le-2.8)=\Phi(-2.6)=0.00466,

and at look 2,

P(Z_2\ge2.0)=1-\Phi(2.4)=0.00820,\qquad P(Z_2\le-2.5)=\Phi(-2.1)=0.0179.

The change in direction lowers the upper-tail marginal probability and raises the lower-tail marginal probability at both looks. However, these four numbers for either effect cannot be summed to obtain the overall probability of stopping: the look statistics are dependent, and paths crossing at look 1 are not continuing paths at look 2.

How to Assess Sensitivity Without Changing the Rule

A useful sensitivity assessment changes only the assumed effect scenario and recalculates the relevant operating quantities under the same information schedule and boundary values. Examine effects near zero and effects in both clinically relevant directions. Small changes can matter most when the mean is near a boundary on the standardized scale, because a modest shift in the normal mean can noticeably change a tail probability.

Freeze the design.
Record \(I_k\), \(u_k\), and \(l_k\) for every look. Do not reflect or move the boundaries when changing the assumed effect.

Specify effect scenarios.
Choose a central value and modest alternatives in both directions, on the same effect scale used to define the canonical drift.

Calculate and label the quantities.
Use the normal tail formulas for stagewise marginal probabilities, or a joint path calculation for first-stopping and overall probabilities. State which type of probability is reported.

Interpret the asymmetry.
Relate changes to the distances from the shifted mean to each fixed boundary and to the intended benefit-harm priorities. Separate operating behavior from claims about the effect actually present.

Diagnostics include checking that the assumed mean at each look is \(\theta\sqrt{I_k}\), that effect and information units are consistent, and that each probability is attached to the correct boundary and direction. A table in which both upper- and lower-crossing probabilities rise as \(\theta\) increases is a warning to recheck signs, tail formulas, or labels. Also check whether the conclusions depend strongly on the normal approximation or on an assumed information schedule that may not hold in practice.

Alternatives, Limitations, and a Common Error

There are two distinct responses if the fixed rule behaves undesirably under plausible effect directions. One is to retain the prespecified rule and report its sensitivity transparently; this is appropriate when the rule is already fixed and the purpose is interpretation. The other is to redesign or recalibrate boundaries before the study, for example to change directional error allocation or to use more symmetric boundaries. That is a design decision, not a sensitivity calculation. Any revised rule needs its own error-rate and power assessment; it cannot be presented as though it were the original rule.

The calculations also have limits. Marginal normal probabilities do not establish overall type I error, power, or expected sample size for a multistage design. Those quantities depend on the full stopping rule and the joint path distribution. Further, a sensitivity analysis over selected assumed effects does not guarantee robustness to misspecified variance, nonproportional effects, delayed effects, or changes in recruitment and information accrual.

Common mistake: “The positive and negative effects have equal magnitude, so their directional crossing probabilities must match after swapping upper and lower.”
Why it is wrong: Sign reversal reflects a boundary at \(u_k\) to \(-u_k\), but an asymmetric rule need not have \(l_k=-u_k\). The compared tail thresholds are then different.
Correction: Keep the original \(u_k\) and \(l_k\), shift the distribution mean for each assumed effect, and calculate each tail probability against its actual boundary. If comparing reflected scenarios, state the boundary reflection required for the equality.

Key takeaway: A fixed asymmetric rule has fixed thresholds, while an assumed effect direction shifts the sequential statistic’s distribution. Direction sensitivity is the resulting change in operating behavior—not a change in the rule—and stagewise marginal crossings must not be mistaken for overall stopping probabilities.

Check Your Understanding

Answer each question using the fixed-boundary interpretation. Solutions follow the questions.

At a look with \(I=36\), \(u=2.0\), \(l=-2.5\), and \(\theta=0.10\), what is the mean of \(Z\), and what is the marginal upper-boundary crossing probability?
Why does \(P_{-\theta}(Z_k\ge u_k)=P_{\theta}(Z_k\le -u_k)\) not generally equal the lower-boundary probability under \(\theta\)?
For the two-look example, why is it incorrect to add the look-1 and look-2 upper-boundary marginal probabilities?
If the assumed effect increases while the boundaries remain fixed, what should happen to the marginal upper- and lower-tail crossing probabilities under the canonical normal model?

Solutions. (1) The mean is \(0.10\sqrt{36}=0.60\), so \(P(Z\ge2.0)=1-\Phi(2.0-0.60)=1-\Phi(1.40)=0.0808\). (2) The reflected threshold is \(-u_k\), whereas the actual lower boundary is \(l_k\); equality requires \(l_k=-u_k\). (3) The statistics across looks are correlated, and a path crossing at look 1 would have stopped rather than continued to look 2; the sum is not a first-stop or overall probability. (4) The distribution shifts right, so the upper-tail probability increases and the lower-tail probability decreases, with boundaries unchanged.