Null Calibration With Discrete Event Counts

Why Null Calibration Can Be Difficult With Discrete Events

A sequential logrank design often specifies efficacy boundaries using a Gaussian approximation to the joint distribution of the logrank statistics. Under the null hypothesis, a boundary intended to spend a particular amount of type I error has that nominal interpretation only to the extent that the approximation and the information schedule describe the actual score process well. With few events, event contributions are discrete; tied failures can make an increment especially coarse. In addition, the information accumulated by a calendar-time cutoff may differ from its planned value.

The key diagnostic is to compare the nominal null crossing probability implied by the design with the probability produced by the event-level score distribution. This tutorial focuses on that comparison. It does not require that the Gaussian approximation be discarded: rather, it asks when the discrete data and realized information make that approximation questionable.

Event-Level Score Contributions and Information

At look \(k\), let \(U_k\) be the cumulative logrank score and \(V_k\) its estimated variance, also called the information for the score. The standardized statistic is \(Z_k=U_k/\sqrt{V_k}\), when \(V_k>0\). A new event group contributes an increment to both the cumulative score and its variance. Because the number of treated-arm failures in a finite risk set is an integer, the score increment generally has a discrete distribution.

For a simple tied-event calculation, suppose a risk set contains \(n\) individuals, of whom \(n_1\) are assigned to treatment. If \(d\) failures occur together and treatment assignment is exchangeable within this risk set under the null, the treated failure count \(O_1\), conditional on the risk set and \(d\), has a hypergeometric distribution. The logrank score increment is \(X=O_1-dn_1/n\). Its null mean is zero, and its variance is the usual tied-event contribution to logrank information.

This calculation assumes that, under \(H_0\), the treatment labels are exchangeable for the failures within the risk set and that the tied-event variance convention matches the analysis. With \(d=1\), the variance reduces to the familiar single-event contribution. Stratified analyses use the corresponding risk-set contributions within strata and add them. The exact distribution can still be discrete even when its mean and variance are correctly calculated.

Why the Gaussian Null Approximation Can Miss

The canonical approximation represents the cumulative score process as approximately Gaussian, with covariance determined by information. If \(I_1\leq I_2\leq\cdots\leq I_K\) are the information levels at the looks, the canonical null model has \(\operatorname{Cov}(U_i,U_j)=I_i\) for \(i\leq j\). Consequently, the correlation between standardized statistics at looks \(i\) and \(j\) is \(\sqrt{I_i/I_j}\). This structure motivates Gaussian boundary calculations, but it does not make a finite collection of discrete increments exactly normal.

A useful justification for the approximation is that a sum of many suitably small, mean-zero score increments can become approximately normal. The approximation is less credible when there are few increments, when one increment accounts for a substantial part of total information, or when the possible cumulative scores are spaced too far apart relative to a boundary. Correctly computing \(V_k\) is essential, but matching a variance alone cannot remove a discrepancy caused by a coarse score distribution.

The discrepancy need not always be liberal. With eight increments of the same form, \(Z_8=S/\sqrt{8}\). Crossing 1.645 requires \(S\geq1.645\sqrt{8}\approx4.652\), hence \(S\geq6\). This occurs with 7 or 8 positive signs, so the probability is \((\binom{8}{7}+\binom{8}{8})/2^8=9/256\approx0.03516\). Here the same nominal cutoff gives a smaller tail probability than 0.05. Discreteness can thus put the actual tail above or below the nominal target, depending on the event count and the boundary's position relative to attainable scores.

Realized Information and Sequential Calibration

In a sequential design, calibration concerns crossing at any look, not merely the tail probability at the final look. The joint crossing probability depends on how information accumulates between looks. Event-driven monitoring can produce realized information levels that differ from the planned levels because event counts arrive in integer increments, risk-set composition changes, or tied events add information in a jump. If the analysis uses boundaries or spending calculations based on planned information fractions, the realized joint null distribution may not match the one used to select those boundaries.

A practical assessment separates two questions: are the observed information levels close to those anticipated, and is the Gaussian approximation adequate for the event-level increments? A close match on the first question does not guarantee the second. Conversely, using realized information in a canonical calculation can improve alignment of the covariance structure, but it does not turn discrete increments into normal ones.

Ways to Assess Calibration and Their Limits

Three approaches are useful, with different strengths. The Gaussian canonical calculation is efficient and is often appropriate when there are many events and no unusually influential increments. Exact enumeration can calculate a null crossing probability for a small, fully specified design when the set of possible event allocations and risk-set updates is manageable. Monte Carlo simulation can approximate the crossing probability under a specified null event process when exact enumeration is too large; its reliability depends on correctly representing randomization, censoring, strata, event-time ties, and the monitoring rule.

For Monte Carlo, if \(B\) independent null trials produce \(R\) boundary crossings, the estimated crossing probability is \(R/B\). Its approximate simulation standard error is \(\sqrt{(R/B)(1-R/B)/B}\); this measures Monte Carlo error, not uncertainty about whether the null model reflects the intended trial. Any simulation should reproduce the actual analysis rule, including when looks occur and how ties and information are handled.

Useful warning signs include a small total number of events, large tied-event groups, a single increment that is large relative to total \(V_k\), realized information that jumps past a planned target, and null simulations whose crossing frequency differs meaningfully from the nominal level beyond simulation error. These are prompts for closer assessment, not automatic proof that a design is invalid. Exact or simulated calibration also has limitations: it is conditional on the risk-set and event-generation assumptions supplied to it, and a precise computation under a misspecified model can still be misleading.

Common Pitfall

A common mistake is to say, “The logrank statistic has variance one after standardization, so the nominal 5% Gaussian boundary must give a 5% type I error.” Unit variance does not determine tail probabilities, especially for a discrete distribution. In the four-event example, \(Z_4\) has variance one, but the probability of crossing 1.645 is 0.0625. The correction is to distinguish the statistic's variance calculation from the approximation used to turn that statistic into a boundary-crossing probability. Check the score support, realized information, and sequential crossing rule under the null.