Directional Consistency Checks for Interim Effects

Why Check Direction as Well as Size?

An interim selection rule may retain an experimental arm because its score, estimated effect, or conditional power exceeds a prespecified threshold. But a sufficiently large score does not always mean that the interim evidence points in the favorable direction. This can happen, for example, when the score uses an absolute effect size or when different interim summaries conflict. A directional consistency check adds a prespecified requirement that the estimated effect point in a clinically favorable direction, or that evidence across stages not show a specified pattern of reversals.

The check is a screening component of an arm-selection rule. Passing it does not establish efficacy, and failing it does not prove that an arm is ineffective. Its role is narrower: prevent an arm from continuing when its interim evidence conflicts with a prespecified directional requirement. The requirement should be fixed before examining the interim results, because choosing a favorable direction or consistency condition after seeing the data changes the operating characteristics of the rule.

The sign convention is part of the rule, not a cosmetic choice. For a continuous outcome where higher values are better, the effect can be treatment minus control. For an outcome where lower values are better, the effect should be coded in the opposite direction, or the favorable sign condition should be reversed. For a binary outcome, a benefit-oriented effect might be a reduction in risk, coded as control risk minus treatment risk. State the outcome, contrast, and favorable direction explicitly.

Common Forms of a Directional Gate

A simple gate checks the sign of the current cumulative effect estimate: \(D_j=I(\hat{\theta}_{jK}>0)\), where \(K\) is the current look and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. This allows an arm to continue only if its current estimate favors treatment. A stricter version requires favorable signs at every observed look: \(D_j=I(\hat{\theta}_{j1}>0,\ldots,\hat{\theta}_{jK}>0)\). This is a check on the sequence of cumulative estimates; it is not the same as requiring every new stage’s data to show benefit.

A stage-increment gate instead evaluates effect estimates from the data newly accrued since the preceding look. For example, it may require every stage-specific estimate to be nonnegative, or may allow a limited unfavorable deviation, such as \(\hat{\theta}^{\mathrm{inc}}_{jk}\geq -m\) for a prespecified tolerance \(m\geq 0\). This can directly screen for conflicting stage-specific evidence, but it may be stringent when each increment is imprecisely estimated. The rule must specify whether zero passes, how ties are handled, which estimates are checked, and any tolerance.

A gate can be paired with many base selection criteria. The score \(R_j\) might be a signed estimated effect, an absolute effect-size score, a rank, or a conditional-power criterion. The gate does not replace the score threshold; it adds a second condition. In particular, if the base score already requires a sufficiently large positive signed effect, a current-sign gate may be redundant. Its value depends on what the score measures and what evidence the rule is intended to screen.

A Short Probability Calculation

A sign gate is not cost-free. Even an arm with a truly favorable effect can have an unfavorable interim estimate due to sampling variability. The following calculation quantifies this for one look under a normal approximation.

Assume the estimate is unbiased and has known standard error \(SE\), so that \(\hat{\theta}\sim N(\theta,SE^2)\). Then a gate requiring \(\hat{\theta}>0\) passes with probability

where \(\Phi\) is the standard normal cumulative distribution function. The equality follows because \((\hat{\theta}-\theta)/SE\) has a standard normal distribution and, by symmetry, \(P(Z>-a)=\Phi(a)\). If \(\theta=0\), the probability of a strictly positive estimate is \(0.5\), assuming a continuous distribution. If \(\theta>0\), it is greater than \(0.5\), but need not be close to 1 when the estimate is imprecise.

For a gate requiring positive estimates from multiple independent stage-specific increments, suppose \(\hat{\theta}^{\mathrm{inc}}_k\sim N(\theta_k,SE_k^2)\) independently across stages. The probability that all \(K\) estimates are positive is then

Under a null effect in every stage, this probability is \(2^{-K}\) under these assumptions. This illustrates why a repeated sign requirement can be much stricter than a single current-sign check. The product formula does not apply automatically to cumulative estimates, which are generally dependent because later cumulative estimates reuse earlier observations. Nor does this calculation establish the overall error rate of a MAMS design.

Applying a Trend Check Across Looks

A trend requirement should specify what “consistent” means. One transparent option is to require all cumulative estimates observed so far to have the favorable sign. Another is to check stage-specific increments, which asks whether each new block of data points in the favorable direction. These rules answer different questions: cumulative estimates summarize the evidence available by each look, whereas increment estimates isolate the information added at each stage.

Design Choices, Diagnostics, and Alternatives

A directional gate is most useful when its definition is tied to a clear concern about the interim evidence. Before implementation, specify the favorable sign, which looks or increments are checked, the treatment of exact zero, any tolerance, and how the gate combines with the base score. Then examine plausible effect patterns, including a consistently favorable effect, a true null, an effect that changes direction, and a favorable current estimate after an unfavorable early look. The gate can substantially alter which arms continue.

Diagnostics should focus on whether the rule behaves as intended, not on whether the observed data satisfy a preferred pattern. Check the effect coding against the clinical outcome definition; verify that the estimates use the planned analysis and information available at each look; and investigate whether an unexpected reversal could reflect sparse data, unstable estimates, or a data-processing problem. A sign flip near zero may be ordinary sampling noise, so report the magnitude and uncertainty as well as the sign. A gate based only on the sign discards that distinction.

Compared with an effect-size threshold alone, a directional gate can screen out a large but unfavorable estimate when the score uses magnitude rather than signed benefit. Compared with a current-sign gate, a repeated-look or increment gate is stricter and can respond to earlier or stage-specific conflict. A signed p-value or conditional-power criterion may already encode direction, depending on how it is defined, but it answers a different question from a separate consistency requirement. None of these selection criteria should be interpreted as a stand-alone confirmatory test.

The gate has important limitations. Early estimates may be noisy, and a strict requirement can discard an arm with a genuinely useful effect. Requiring every stage-specific estimate to be favorable is especially sensitive to random variation and can penalize later stages with little information. Conversely, a current-sign gate may ignore a substantial earlier reversal. Selection thresholds and directional gates may also affect subsequent inference; their use does not by itself guarantee control of type I error or familywise error. Those properties must be addressed in the design’s inferential procedures, not assumed from a sign check.