Simulating Monitoring Designs with Delayed Effects

Why Simulate a Monitoring Design?

With a delayed treatment effect, the treatment and control hazards may be similar early in follow-up and differ later. Consequently, the timing and weighting of interim analyses can matter: an early look may contain many events but little information about a benefit that has not yet emerged. Simulation makes the full sequence explicit. It follows hypothetical participants from randomization through event or censoring, applies the planned analyses at their scheduled looks, and records whether and when each trial stops.

The aim is not to replace the design’s prespecified analysis or boundary calibration. It is to check how the complete monitoring procedure behaves under specified survival and censoring assumptions. A valid simulation must reproduce the design as written, including the event-time model, censoring, weights, information targets, and stopping rules.

Specify the Trial Before Generating Data

Begin by stating the assumptions and design choices that determine one simulated trial. For example, suppose participants are assigned equally to treatment and control, the control event time is exponential, and the treatment hazard is piecewise constant with a delayed effect. Let \(\lambda_0\) be the control hazard, \(\tau\) the delay, and \(r\) the treatment-to-control hazard ratio after the delay. Under this model,

Here \(\lambda_1(t)\) is the treatment hazard. The model says there is no treatment effect before \(\tau\), followed by a lower treatment hazard thereafter. It is a convenient hypothetical model, not a claim that real treatment effects must have this form. Other piecewise hazard specifications are possible, but they must be set before interpreting the resulting simulation.

To generate an event time, draw \(U\sim\text{Uniform}(0,1)\) and set \(Q=-\log(U)\). An event occurs when the cumulative hazard reaches \(Q\). For a participant assigned to treatment in the model above, the event time \(T\) is

For control, \(T=Q/\lambda_0\). Censoring is generated separately. If \(C\) is a censoring time, the observed follow-up is \(X=\min(T,C)\), and the event indicator is \(\delta=I(T\leq C)\). A common assumption for this simulation is that censoring is independent of event time, conditional on treatment arm and any modeled covariates. Administrative censoring at a common cutoff can be combined with independently generated loss-to-follow-up time by taking their minimum. The censoring model should reflect the design assumptions; it should not be chosen merely to produce a desired number of events.

The monitoring plan also needs to be specified: how many looks there are, what triggers each look, the weights used in the weighted logrank statistic, and the efficacy and any futility boundaries. A look might be triggered by a cumulative event target, or by a prespecified information target. Information time is based on the statistic’s variance, not automatically on calendar time or event count.

Calculate the Interim Statistic with a Consistent Sign

For each event time \(t_j\) at a look, let \(Y_{1j}\) and \(Y_{0j}\) be the numbers at risk in the treatment and control arms just before the event, and let \(d_{1j}\) be the number of treatment events at that time. With no tied event times, \(d_j=1\). For prespecified nonnegative weights \(w_j\), define the weighted logrank score as observed minus expected treatment events:

In this convention, fewer treatment events than expected make \(U_k\) negative. That is the direction expected when treatment lowers the hazard. Therefore define the treatment-favoring standardized statistic as \(Z_k=-U_k/\sqrt{V_k}\): positive values point toward treatment benefit. This sign choice must agree with the efficacy boundary. If a design instead uses \(U_k/\sqrt{V_k}\), its treatment-benefit boundary must be negative.

For tied events, the score uses the observed number of treatment events minus its risk-set expectation, and the variance uses the corresponding hypergeometric variance, with the weights squared. The formula above is the no-tie version, appropriate when event times are continuous or ties are negligible. The same tie method must be used in design calibration and simulation analysis.

The earlier material on independent increments and information time motivates treating \(V_k\) as the accumulated variance, or information, for this score. Under the null and the usual large-sample conditions, standardized increments are approximately normal. That approximation supports sequential boundary methods, but it does not excuse changing the statistic, weights, or look schedule after calibration.

Apply the Monitoring Rules in Each Trial

At each scheduled look, compute the statistic using all eligible data accumulated by that look. Compare it with the prespecified boundaries. For a one-sided efficacy test of treatment benefit, an upper boundary \(b_k\) is crossed when \(Z_k\geq b_k\). A futility rule might stop when \(Z_k\leq f_k\), where \(f_k\) is a lower boundary. Whether futility stopping is binding or nonbinding must be defined in advance: a binding rule changes which paths can proceed to later looks.

The boundaries must correspond to the same analysis, weights, and monitoring plan used in the simulation. They may come from a prespecified alpha-spending design or another calibrated sequential procedure. If a statistic is changed from \(U/\sqrt V\) to \(-U/\sqrt V\), the boundary directions must be changed consistently. Do not treat illustrative boundary values as calibrated values.

Summarize Replicates and Check the Simulation

For \(R\) simulated trials, estimate an operating characteristic by the corresponding proportion. For example, if \(R_E\) trials cross an efficacy boundary, the estimated probability of efficacy stopping under that scenario is \(\hat p_E=R_E/R\). Its approximate Monte Carlo standard error is \(\sqrt{\hat p_E(1-\hat p_E)/R}\). This quantifies simulation noise, not uncertainty about whether the assumed model is scientifically correct.

A basic implementation should be checked before interpreting its results. Verify that generated event times are positive, that pre-delay and post-delay cumulative hazards follow the specified model, and that every censored observation has \(\delta=0\). Check that risk sets decrease appropriately, variance contributions are nonnegative, and \(V_k\) does not decrease across looks. Under the null model, the estimated probability of efficacy crossing should be compared with the intended one-sided type I error, allowing for Monte Carlo uncertainty. Also inspect stopping-look frequencies and the distribution of event counts; plausible overall rates can conceal coding errors in the interim sequence.

What Simulation Can and Cannot Establish

Simulation is particularly useful for checking a whole monitoring path under a delayed-effect model, including censoring and stopping. It complements analytic approximations based on standardized sequential statistics: analytic methods can be efficient for calibration under their assumptions, whereas trial-level simulation can represent event-time generation and censoring directly. Neither approach repairs a mismatch between the planned and implemented statistic.

A simulation result is conditional on its inputs. It cannot establish that the assumed delay, hazard shape, censoring distribution, or treatment effect is correct. Results should therefore be reported for clearly stated scenarios, and important assumptions can be varied in sensitivity analyses. The main limitations are computational cost, Monte Carlo variability, and the possibility of a precise answer to a poorly specified model.

Common Mistakes and Failure Modes

• Reversing the score direction. With \(U=\) observed minus expected treatment events, fewer treatment events give \(U<0\), the direction of benefit. Use \(Z=-U/\sqrt V\) for positive treatment-favoring values, or keep the opposite sign and reverse the boundary direction.
• Calling a favorable sign significant. A statistic in the benefit direction need not cross an efficacy boundary. Compare it with the correct boundary for that look.
• Using censoring as if it were an event. Only observed events contribute as events to the logrank score. A censored participant leaves the risk set at censoring without an event contribution.
• Changing weights after seeing simulated outcomes. Data-dependent adaptation can alter the null distribution. Use weights and any adaptation rule that were prespecified and calibrated for the analysis.
• Mixing monitoring definitions. Event-count looks, variance-based information looks, and calendar-time looks need not occur at the same times. Implement the planned trigger exactly.
• Confusing simulation precision with model validity. More replicates narrow Monte Carlo error but do not make the assumed hazards or censoring mechanism more realistic.