A Two-Look Pocock Calibration Example

Why Work Through a Two-Look Calibration?

A Pocock design uses the same efficacy cutoff at each planned analysis. That common cutoff is not simply the ordinary final-analysis critical value: looking more than once creates more than one opportunity to cross a boundary. Calibration must account for the chance of crossing at either look under the null hypothesis. For two looks, the calculation is small enough to inspect directly, making it a useful way to connect the information schedule, the dependence between interim statistics, and the overall type I error.

This example concerns a planned two-sided design with familywise type I error \(\alpha=0.05\), two analyses at information fractions \(0.5\) and \(1\), and no other stopping rules. The Pocock rule rejects \(H_0\) if the absolute value of either cumulative Z statistic reaches a common cutoff \(c\). The calculation uses the null joint distribution of those statistics, not an assumption that the two looks are independent.

Information Fractions Determine the Correlation

Let \(I_1\) and \(I_2\) be the planned information at the first and second looks, with \(0<I_1<I_2\). Their information fractions are \(t_1=I_1/I_2\) and \(t_2=1\). In a canonical Z-statistic design under \(H_0\), the cumulative score at look 2 contains the score accumulated by look 1 plus the new information accumulated between looks. With independent increments, the score variances are \(I_1\) and \(I_2\), and the covariance of the two cumulative scores is \(I_1\). Standardizing each score gives

Thus \((Z_1,Z_2)\) has a bivariate standard normal null distribution with correlation \(\rho=\sqrt{t_1}\). This result assumes the planned information is correctly specified, the cumulative statistics have the canonical normal form under \(H_0\), and information is accumulated through independent increments. It does not say the statistics are identical: the later statistic includes new data. Nor does it say they are independent; for an interim look before the final analysis, their correlation is positive.

Calibrating the Common Cutoff

For a two-sided design, the null event of no rejection is that both statistics lie strictly between \(-c\) and \(c\). Consequently, the familywise type I error is the probability of the complementary event: at least one of the two statistics has absolute value at least \(c\). Because the statistics are correlated, this probability must be calculated from their joint distribution.

For standard normal variables with correlation \(\rho\), the conditional distribution of \(Z_2\) given \(Z_1=z\) is normal with mean \(\rho z\) and variance \(1-\rho^2\). Integrating the conditional probability that \(Z_2\) is inside the cutoff over values of \(Z_1\) inside the cutoff gives the probability of no crossing:

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. The calibration sets one minus this integral equal to the target \(\alpha\). In general, the integral is evaluated numerically and the cutoff is found by numerical root-finding. The dependence matters: replacing the joint probability with a calculation that treats the looks as independent will not give the correct calibration.

Applying the Calibrated Rule

Calibration determines the cutoff before outcomes are examined. At analysis time, the decision is made by comparing each observed statistic with that cutoff, in the direction or directions specified by the design. For a two-sided design, the sign of the statistic indicates the direction of the observed effect, while the absolute value determines whether the boundary has been crossed.

Derivation, Diagnostics, and Alternatives

The key justification for the correlation calculation can be seen by writing the cumulative scores explicitly. Let \(S_1\) be the score accumulated by the first look and let \(S_{\mathrm{new}}\) be the additional score between looks. Under the canonical null model, assume these increments are independent, with variances \(I_1\) and \(I_2-I_1\). Then \(S_2=S_1+S_{\mathrm{new}}\), so

Since \(Z_1=S_1/\sqrt{I_1}\) and \(Z_2=S_2/\sqrt{I_2}\), standardization yields \(\operatorname{Corr}(Z_1,Z_2)=I_1/\sqrt{I_1I_2}=\sqrt{I_1/I_2}\). The common cutoff then follows by applying the probability of the union of the two crossing events, rather than adding two marginal probabilities as if the events were independent.

In practice, check that the planned information fractions correspond to the actual analysis schedule and that the analysis statistics are compatible with the assumed canonical joint null distribution. If the interim information differs materially from plan, the correlation implied by the planned fraction may no longer describe the data. Likewise, changes in the statistic, variance estimation, or dependence structure can require an appropriate recalculation. Do not assume that the square-root correlation formula applies merely because analyses are labelled “interim” and “final.”

A common mistake is to use the ordinary two-sided final-analysis cutoff \(1.96\) at both looks because it corresponds to a single-analysis \(\alpha=0.05\). This ignores the chance of crossing at the interim look as well as at the final look, so it does not provide the stated familywise error control. A different shortcut is Bonferroni allocation: assign a two-sided error probability \(0.025\) to each look, giving a per-look cutoff of approximately \(2.241\). This controls the familywise error by the union bound but is generally conservative because it does not use the correlation. Pocock calibration instead finds a common cutoff from the joint null distribution. Other sequential designs can use different cutoffs across looks; their calibration and boundary shapes are distinct from the common-cutoff calculation here.

The result is specific to its assumptions and plan. It is not a universal cutoff for all two-look designs: the cutoff depends on \(\alpha\), the information fraction, whether the test is one-sided or two-sided, the null distribution, and the stopping rule. An additional rule that stops for futility or changes the analysis schedule must be described when calculating the operating characteristics; the simple equation above represents the stated two-look efficacy rule.