Evaluating Delayed or Incomplete Interim Data

Why Interim Data Availability Is Part of the Design

An adaptation rule may be specified precisely and still behave differently in operation if the records it needs are not ready when the decision is scheduled. An outcome may require a fixed follow-up period, a site may submit records late, or a participant’s record may be only partially complete. The decision then uses the data available at a particular time, not the data that will eventually be available for the full trial.

In design assurance, the data-availability process should therefore be represented alongside the outcome model and adaptation rule. The key question is not only whether the planned rule performs well with complete interim data. It is whether it performs acceptably under plausible patterns of delay and incompleteness, and whether those patterns change trial duration or final operating characteristics.

These are operational states, not by themselves explanations for why data are unavailable. A simulation should specify how availability evolves over time and, where relevant, how that process relates to outcomes or participant characteristics. It should also represent the exact data-cutoff rule: for example, whether a record must be received and verified by the cutoff, or whether a visit completed by the cutoff is sufficient.

Representing Availability in a Simulation

For simulated participant \(i\), let \(Y_i\) denote the outcome defined by the trial’s analysis plan, and let \(R_i(t)\) indicate whether the outcome information required by the interim rule is available at time \(t\). A simple indicator takes value 1 when the required information is available and 0 otherwise. The adaptation rule is applied to the available records, not to outcomes that would only become known later.

A defensible simulation specifies at least four components: the outcome-generating model, the time at which each record becomes available, the decision cutoff and eligibility rules, and the adaptation rule applied to the resulting interim data. If availability depends on a participant’s outcome, site, treatment group, or follow-up history, the simulation needs to represent that dependence in scenarios of interest. Assuming independent delays is a scenario assumption, not a default truth.

For a decision rule \(g\), the probability of an interim action must average over both the generated outcomes and the generated availability pattern. Write \(R\) for the full availability pattern at the decision time. Then:

Here \(\theta\) denotes the scenario’s data-generating parameters. This expression emphasizes a practical point: even if the outcome model is unchanged, altering the distribution of \(R\) can alter the probability of an action. The final operating characteristics must then be calculated by carrying that action forward through the rest of the simulated trial.

Why Fewer Available Records Can Change a Trigger

If available outcomes are a representative subset and the outcome variance is unchanged, fewer observations generally make an interim estimate less precise. For two independent groups with a common outcome standard deviation \(\sigma\), available sample sizes \(m_1\) and \(m_0\) give the standard error:

This follows because the variance of a sample mean based on \(m_j\) independent observations is \(\sigma^2/m_j\), and the variance of the difference of independent means is the sum of those variances. Reducing either available sample size increases the standard error. But this calculation describes precision under its assumptions; it does not correct bias if the available records are a selected subset. Nor does it account for interim rules that use other statistics, multiple outcomes, or a model for partial follow-up.

Selection and Partial Follow-Up Need Separate Scenarios

A smaller available sample is not always merely a less precise version of a complete sample. If the records available at the cutoff differ systematically from those not yet available, an estimate based on available records can shift as well as become less precise. For example, participants with more complex follow-up may have slower record completion, and their outcomes may differ from those with prompt records. The direction and size of the resulting shift depend on the scenario; there is no universal correction based only on the percentage of records missing.

Partial follow-up needs particular care. If an endpoint is defined at a fixed follow-up time, an early measurement is not automatically that endpoint. Substituting an interim measurement into the final endpoint rule changes what is being measured. A design may instead specify a valid model using partial information, but then the model, its assumptions, and the data-availability rule all belong in the simulation. Do not treat an unobserved outcome as if it were observed, or silently treat a short follow-up value as a completed endpoint.

Simulate the Decision Timeline, Not Just the Decision Statistic

A delay can affect calendar duration even when it does not change the final number of participants. The effect depends on the operational policy. Recruitment might pause while the trial waits for interim outcomes, continue while the review is pending, or continue only up to a prespecified limit. These alternatives can lead to different numbers of participants recruited before the decision, different exposure to an eventual expansion or stopping action, and different end dates.

For each simulated trial, track when participants enter, when their required outcomes become available, when the interim dataset is cut, when the review occurs, and when the final endpoint is available. Apply the same timing and eligibility rules that operations would use. Summarize both the adaptation outcomes and the elapsed calendar time. A simulation that counts only participants and ignores when data become available cannot quantify delays in decision-making or completion.

What to Record and How to Diagnose Problems

For every simulated replicate, retain enough information to reconstruct what the rule could have known. Useful outputs include the number randomized, number eligible at each interim cutoff, follow-up time represented, count and reason for records not available, interim statistic, action taken, and calendar time of decision and trial completion. These records make it possible to distinguish a design effect from an implementation error.

Useful summaries include the probability of each interim action, final sample size, duration, and the final operating characteristics under the stated scenario. Compare these with a complete-and-on-time reference simulation to quantify the effect of the availability assumptions. Paired simulation—using the same underlying generated outcomes in the reference and delayed-data versions—can make that comparison more precise, provided the paired scenarios are constructed consistently.

• Decision changes sharply with modest changes in delay. Check whether many interim statistics lie close to the trigger and whether the cutoff or eligibility rule is implemented correctly.
• Available records have different group sizes or follow-up times. Verify that the interim statistic and its standard error match the data actually used.
• Action probabilities barely change despite substantial delays. Check that delayed records truly affect the simulated decision input; the code may be accidentally using eventual outcomes.
• Duration is unchanged when a pause is specified. Audit the event timeline and confirm that recruitment actually stops until the interim data are eligible.
• Final summaries use different participants or endpoints across scenarios. Check that the final analysis population and endpoint definition remain those in the design specification.

Comparisons and Limits

A complete-data interim analysis is a useful benchmark: it isolates the consequences of acting as if every required outcome were promptly available. It is not a substitute for a delay scenario when the real decision will use an incomplete dataset. A fixed calendar cutoff is another policy: it makes the decision date predictable but may leave fewer eligible records. Waiting for a fixed number of eligible outcomes may provide a more stable interim sample, but the decision date then varies and can extend the trial. Continuing recruitment during the wait can reduce calendar delay, while also allowing more participants to enter before the adaptation is applied.

None of these policies is uniformly best. Their consequences depend on the outcome latency, availability pattern, operational constraints, adaptation thresholds, and the value placed on timely decisions versus more complete interim information. Simulation results are conditional on the scenarios supplied. If the distribution of delays or the relationship between availability and outcomes is misspecified, the results may not describe operational performance. Sensitivity analysis should therefore include plausible alternative availability patterns, including unfavorable ones, rather than relying on a single assumed delay.