Discrete Look Times and Boundary Crossing

Why the timing of an analysis matters

In a multi-arm trial, treatment arms need not have outcomes available at the same time. One arm might reach its planned information level before another, so its test statistic is examined first. Familywise error depends on which statistics are actually examined, when they are examined, and how their joint distribution is modelled. Treating an arm as continuously monitored when it is only analysed at scheduled looks changes the rejection event—and can misstate the error rate.

Here, a planned look is a prespecified analysis at which the relevant statistic is calculated and compared with a boundary. Crossing a boundary means that the statistic meets or exceeds its rejection threshold at one of those analyses. A statistic that would have crossed between planned looks does not cause rejection under a discrete-look design, unless the protocol specifies an additional analysis or another action at that time.

Definition: Suppose $m$ treatment hypotheses have arm-specific planned look times $t_{i1}<\cdots<t_{iK_i}$, and let $Z_i(t_{ik})$ be the test statistic for arm $i$ at its $k$th planned look. With upper rejection boundaries $b_{ik}$, the discrete-look familywise error rate is $$ \operatorname{FWER} = P\!\left(\bigcup_{i\in I_0}\ \bigcup_{k=1}^{K_i} \{Z_i(t_{ik})\ge b_{ik}\}\right), $$ where $I_0$ is the set of truly null treatment hypotheses. Strong familywise error control at level $\alpha$ means this probability is at most $\alpha$ for every configuration of true and false null hypotheses.

The times $t_{ik}$ may be recorded on a calendar scale, an information scale, or both. A calendar time describes when data become available; an information time describes the amount of information in the statistic. For error calculations, the schedule alone is not enough: the joint null distribution of all statistics at the scheduled looks is also needed. Shared control groups, overlapping data, and repeated analyses can make the statistics dependent.

A useful working model is that the vector of scheduled statistics under a specified null configuration has a known or adequately approximated multivariate normal distribution. Its means are zero for true null hypotheses, and its covariance matrix must reflect the design and information accumulated by each look. This model is not universal: if the normal approximation or covariance assumptions are inadequate, error should be evaluated using a more appropriate distribution or a design-based simulation. Analyses must also follow the specified schedule; unplanned looks or data-dependent timing require their own justification.

Calculating error on the scheduled grid

When the joint distribution is specified, discrete-look FWER is the probability of a union of boundary-crossing events over the planned arm-look pairs. It can be calculated using joint tail probabilities, recursive probability calculations, or simulation. For asynchronous schedules, include each arm’s actual planned looks; do not insert a statistic for an arm at a time when that arm is not analysed.

Worked Example: Two Independent Arms Analysed at Different Times

Suppose two independent treatment statistics each have one planned, one-sided test. Arm A is analysed at week 8 and Arm B at week 12. Under either arm’s null hypothesis, its statistic is standard normal. Each arm has boundary 2.326, which gives a marginal upper-tail probability of approximately 0.0100.

Because the two statistics are independent, the probability of no false rejection is the product of their non-rejection probabilities: $$ P(\text{no false rejection})=(1-0.0100)(1-0.0100)=0.9801. $$ Therefore, $$ \operatorname{FWER}=1-0.9801=0.0199. $$

The familywise error is 0.0199, or 1.99%, across these two true null hypotheses. The fact that the looks occur in different weeks does not change this result under the stated independence and marginal-distribution assumptions. If a look at week 8 led to a decision that altered the later design, that adaptation would need to be represented in the error calculation; it is not part of this fixed-schedule example.

For a single arm with repeated looks, the statistics are generally correlated because later analyses include information from earlier data. A canonical illustration uses a standard Brownian motion $B(t)$ and $Z(t)=B(t)/\sqrt{t}$. Under the null, $Z(t)$ has mean zero and variance one, and for $s\le t$, $$ \operatorname{Cov}\{Z(s),Z(t)\} = \frac{\operatorname{Cov}\{B(s),B(t)\}}{\sqrt{st}} = \frac{s}{\sqrt{st}} = \sqrt{\frac{s}{t}}. $$ This covariance describes one common-information model; it should not be imposed automatically when arms have different information processes or the design induces a different covariance structure.

Worked Example: One Arm with Two Discrete Looks

Consider one true null hypothesis with looks at information times $t=0.5$ and $t=1$, and a boundary of 2.000 at each look. Assume the canonical Brownian model just described. The two statistics are standard normal with correlation $\sqrt{0.5}=0.7071$. A false rejection occurs if either statistic exceeds 2.000.

Each marginal upper-tail probability is $P\{N(0,1)\ge 2.000\}\approx0.02275$. The joint probability of exceeding the boundary at both looks is $$ \int_{2}^{\infty}\phi(x)\, \left[1-\Phi\!\left(\frac{2-\sqrt{0.5}\,x}{\sqrt{0.5}}\right)\right]\,dx \approx 0.00748, $$ where $\phi$ and $\Phi$ are the standard normal density and distribution functions. Thus inclusion-exclusion gives $$ \operatorname{FWER} =0.02275+0.02275-0.00748 \approx 0.0380. $$ The chance of at least one boundary crossing at the two scheduled looks is approximately 3.80%. Adding the marginal probabilities without subtracting their overlap would give 4.55%, a conservative upper bound rather than the exact answer.

Discrete looks are not continuous monitoring

In the preceding example, a continuous-monitoring rule would reject if $Z(t)$ reached 2.000 at any time in the interval from 0.5 to 1, not only at its endpoints. The discrete rejection event is contained in that continuous event: a crossing at either scheduled look is also a crossing at some time in the interval. Consequently, under the same process and boundary, $$ P(\text{crossing at a scheduled look}) \le P\!\left(\sup_{0.5\le t\le 1} Z(t)\ge 2.000\right). $$ The inequality can be strict because the process can cross and then return below the boundary before the next planned analysis. In this example, the discrete probability is approximately 0.0380; it is not the probability of continuous crossing. A continuous-monitoring probability requires a calculation for the supremum over time, not just the multivariate distribution at the planned looks.

Key distinction: Discrete-look error is calculated over a finite set of scheduled arm-look pairs. Continuous-monitoring error is calculated over every time in the specified monitoring interval. Using a continuous boundary-crossing probability for a discrete design, or a discrete-look probability for a continuously monitored design, answers the wrong probability question.

The event inclusion also clarifies an important limitation: a discrete calculation is valid only if decisions really are made at the stated looks. If investigators inspect accumulating data more often than planned and may stop for efficacy whenever a statistic crosses the boundary, the operational design is closer to continuous or more-frequent monitoring. The planned-look calculation will then omit possible rejection events.

Dependence, strong control, and practical evaluation

The same event-based approach applies when several arms share a control group or have multiple asynchronous looks. For a specified set $I_0$ of true nulls, form the joint null distribution for the scheduled statistics on those arms and evaluate the probability of at least one true-null crossing. To claim strong control, this bound must hold for every possible true-null set, not only when all treatment hypotheses are null. A calculation under the complete null alone does not establish strong control unless an additional valid argument supports that conclusion.

Worked Example: A Conservative Bound with Shared Control

Suppose two arms share a control group and each has one scheduled look. Under each arm’s null, its standard normal statistic is tested against 2.326, so each marginal type I error is approximately 0.0100. The two statistics may be correlated; for example, their correlation might be positive because of the shared control.

Without using the correlation, the union bound gives $$ P(\text{at least one false rejection}) \le P(Z_1\ge2.326)+P(Z_2\ge2.326) \approx 0.0100+0.0100=0.0200. $$ Thus the FWER is no greater than 0.0200 under the stated marginal null distributions, whatever the dependence between the two statistics. This is a valid but potentially conservative bound; it does not assert that the actual FWER equals 0.0200. To obtain a less conservative value, calculate the joint upper-tail probability using the appropriate covariance, then subtract the overlap from the sum of the marginal probabilities.

If the design has repeated or asynchronous looks, the calculation must include all scheduled looks for each true-null arm, with their full joint covariance. A bound that sums marginal error probabilities remains available, but it may be loose when there are many looks.

Several approaches are useful for different purposes. A joint multivariate-normal calculation can be efficient when the scheduled-statistic distribution and covariance are trustworthy. Recursive probability calculations can exploit sequential structure and account for boundary crossing over successive looks. Simulation can check more complicated schedules, dependence patterns, and null configurations, but its accuracy depends on the number of simulated trials and correct implementation of the design. Bonferroni or other valid union bounds require less dependence information but can sacrifice power through conservatism. None of these approaches makes a continuous-monitoring calculation interchangeable with a discrete one.

Diagnostics should target the pieces that define the event and its distribution. Check that the analysis schedule includes every permitted look, that each statistic is paired with the correct boundary, and that the covariance matrix matches the data overlap and information times. In simulation, verify the marginal null rejection probabilities and confirm that recorded rejection occurs only at permitted analyses. Under adaptive selection or dropping, track which hypotheses remain eligible for rejection and ensure the calculation represents the actual decision rule. A numerical answer can be precise while still being wrong if it describes the wrong schedule or null configuration.

Specify the decision rule.
List the scheduled analysis times for each arm and state exactly which boundary crossing causes rejection.

Specify the null configuration and joint model.
Identify the true null hypotheses and the joint distribution of their statistics at all relevant planned looks.

Evaluate the union event.
Calculate or bound the probability of at least one true-null crossing, using joint probabilities, recursion, simulation, or a valid union bound.

Check the scope of the claim.
For strong control, verify the required error bound across true-null configurations; do not infer continuous-monitoring control from a discrete-look calculation.

Common mistake and takeaway

Common mistake: “The statistic could cross the boundary between weeks 8 and 12, so that crossing belongs in the discrete-look FWER.” This is incorrect if no analysis or rejection decision is permitted between those scheduled looks. The correction is to define the rejection event from the actual decision rule: include only scheduled analyses for a discrete design, and include all times in the monitoring interval for a continuous design. Conversely, if investigators are permitted to act on an interim crossing, the design cannot be evaluated as though only the listed discrete looks were observed.

The practical lesson is that familywise error is a probability attached to a complete testing procedure: hypotheses, schedules, boundaries, dependence, and decision rules together. Asynchronous timing does not itself invalidate error control, but it must be represented correctly in the joint crossing calculation.

Check Your Understanding

Answer each question, then compare with the short solutions below.

Two independent true-null tests each have one planned look and marginal type I error 0.025. What is their exact FWER?
For two scheduled looks with marginal crossing probabilities 0.02 and 0.03, what upper bound follows from the union bound, without knowing their dependence?
Why does a discrete-look crossing probability not exceed the corresponding continuous-monitoring crossing probability when the same process and boundary are used?
Does evaluating FWER when all treatment hypotheses are null by itself establish strong control? Explain.

Solutions. (1) Independence gives $1-(1-0.025)^2=1-0.950625=0.049375$, or about 4.94%. (2) The union bound is $0.02+0.03=0.05$, so FWER is at most 0.05. (3) Every crossing at a scheduled look is also a crossing at some time in the continuous interval, so the discrete rejection event is a subset of the continuous rejection event; its probability cannot be larger. (4) No. Strong control requires the specified bound for every configuration of true and false null hypotheses, unless a valid additional argument establishes it from a smaller set of configurations.