Why Cumulative Interim Tests Are Dependent

Why Repeated Interim Tests Are Not Usually Independent

At a planned interim analysis, a trial may test its null hypothesis using everyone enrolled so far. At the next analysis, those same participants remain in the data, joined by additional participants. The later statistic therefore contains part of the random information used in the earlier statistic. This structure can make the statistics dependent—but shared data alone do not prove dependence. The relationship must be checked for the particular statistics and probability model.

This distinction matters when reasoning about repeated tests. The earlier material on the union bound gives an upper bound on the probability of at least one false positive without requiring independence. But to understand the actual joint behavior of cumulative test statistics, we need to examine their covariance under an explicit model.

A Cumulative-Test Model

Suppose observations arrive in a fixed sequence, and the first look occurs after \(n_1\) observations while the second occurs after \(n_2\), with \(0<n_1<n_2\). For a simple one-sample mean test, assume the observations \(X_1,X_2,\ldots\) are independent and identically distributed as \(N(\mu_0,\sigma^2)\) under the trial-wise null hypothesis \(H_0:\mu=\mu_0\). Assume \(\sigma\) is known and the analysis uses the observations accumulated through each look. These assumptions make the calculation exact and transparent; other designs require their own covariance calculation.

Standardize each observation under the null by defining \(Y_i=(X_i-\mu_0)/\sigma\). Then the \(Y_i\) are independent, each has mean 0 and variance 1, and each cumulative \(Z\)-statistic is

Z_j=\frac{\sum_{i=1}^{n_j}Y_i}{\sqrt{n_j}} =\frac{\sqrt{n_j}(\bar{x}_j-\mu_0)}{\sigma}.

Here \(n_j\) is the cumulative sample size at look \(j\), and \(\bar{x}_j\) is the mean of the first \(n_j\) observations. Under the stated normal model and \(H_0\), each \(Z_j\) has a standard normal distribution. That fact describes each statistic separately; it does not imply that statistics from different looks are independent.

Definition: In a cumulative design, a later look includes the observations used at earlier looks, as well as any new observations accrued in the meantime. The test statistics are dependent if their joint distribution does not factor into the product of their separate distributions. In the cumulative normal model above, dependence is established by the nonzero covariance of the statistics.

Deriving the Dependence

For the two-look case, write each statistic as a sum. In the covariance, observations used at both looks contribute; observations available only at the second look do not contribute to the covariance because they are independent of the first-look observations.

\begin{aligned} \operatorname{Cov}(Z_1,Z_2) &= \operatorname{Cov}\left( \frac{\sum_{i=1}^{n_1}Y_i}{\sqrt{n_1}}, \frac{\sum_{i=1}^{n_2}Y_i}{\sqrt{n_2}} \right)\\ &= \frac{1}{\sqrt{n_1n_2}} \sum_{i=1}^{n_1}\operatorname{Var}(Y_i)\\ &= \frac{n_1}{\sqrt{n_1n_2}} = \sqrt{\frac{n_1}{n_2}}. \end{aligned}

The second equality follows from independence across observations: cross-covariances between distinct \(Y_i\)'s are zero, while each shared observation has variance 1. Since both statistics have variance 1, their correlation is also \(\sqrt{n_1/n_2}\). This quantity is positive and less than 1 when \(0<n_1<n_2\).

Under the model, the pair \((Z_1,Z_2)\) is jointly normal because both statistics are linear combinations of independent normal observations. For jointly normal variables, zero covariance is equivalent to independence. Here the covariance is positive, not zero, so the statistics are dependent. For more than two looks, the same argument gives, for \(j<k\),

\operatorname{Cov}(Z_j,Z_k) = \operatorname{Corr}(Z_j,Z_k) = \sqrt{\frac{n_j}{n_k}}.

This expression is specific to the cumulative, independent, equal-variance normal model with known \(\sigma\). It should not be transferred automatically to a different test statistic or data structure.

Worked Example: Two Cumulative Looks

Hypothetical setup. Suppose the first analysis occurs at \(n_1=4\), and the second at \(n_2=9\). Under the stated null model, the correlation is

\operatorname{Corr}(Z_1,Z_2) = \sqrt{\frac{4}{9}} = \frac{2}{3} \approx 0.667.

For an illustration of the observed statistics, suppose the first four standardized observations are \(0.2,-0.4,0.8,0.6\). Their sum is \(1.2\), so \(Z_1=1.2/\sqrt{4}=0.600\). Suppose the next five standardized observations are \(-0.3,0.5,-0.2,0.1,0.4\), with sum \(0.5\). Across all nine observations the sum is \(1.2+0.5=1.7\), giving \(Z_2=1.7/\sqrt{9}\approx0.567\).

These two realized values are not a demonstration of dependence by themselves: one pair of numbers cannot establish a joint probability relationship. The model-based covariance calculation does that. It shows that, across repeated trials under \(H_0\), the two statistics have correlation about 0.667 because four observations contribute to both.

Seeing the Earlier Statistic Inside the Later One

The same dependence can be understood by separating the second-look sum into the first-look observations and the new increment. Under the model, the increment is independent of the first-look data. This gives a useful representation of how the earlier statistic contributes to the later one.

Z_2 = \sqrt{\frac{n_1}{n_2}}Z_1 + \sqrt{\frac{n_2-n_1}{n_2}}W, \qquad W= \frac{\sum_{i=n_1+1}^{n_2}Y_i}{\sqrt{n_2-n_1}}.

The increment statistic \(W\) has mean 0 and variance 1 under \(H_0\), and is independent of \(Z_1\), because it uses only observations that arrive after the first look. The first term in the representation carries forward the earlier statistic. The second term adds new information. This representation also checks the variance: the two terms are independent, so their variances add to \(n_1/n_2+(n_2-n_1)/n_2=1\).

Worked Example: Decomposing the Later Statistic

Hypothetical setup. Let \(n_1=36\) and \(n_2=64\), so there are 28 new observations between the looks. Suppose the observed first-look statistic is \(Z_1=1.20\), and the standardized sum of the 28 new observations is \(-2.1166\), giving \(W=-2.1166/\sqrt{28}\approx-0.400\). Then

\begin{aligned} Z_2 &= \sqrt{\frac{36}{64}}(1.20) + \sqrt{\frac{28}{64}}(-0.400)\\ &= 0.750(1.20)+0.6614(-0.400)\\ &= 0.900-0.2646 = 0.6354. \end{aligned}

The later statistic retains a contribution from the first-look data, while the new observations also affect its value. Under this model, the correlation between the two statistics is \(\sqrt{36/64}=0.750\). A large first statistic does not force a large second statistic: the independent increment may increase or decrease it.

When Shared Data Do and Do Not Establish Dependence

The cumulative-sum derivation shows exactly why dependence occurs here: shared observations make a positive contribution to the covariance, and the specified normal model makes nonzero covariance decisive for dependence. It would be incorrect to generalize that result to any pair of quantities that happen to use the same data. For example, if \(X\sim N(0,1)\), then \(\operatorname{sign}(X)\) and \(|X|\) both use \(X\), but they are independent. Sharing random input can induce dependence; whether it does so depends on the statistics and their model.

Worked Example: Contrast With Independent Samples

Hypothetical comparison. Suppose two one-sample \(Z\)-statistics are calculated from disjoint groups of independent observations: the first group has 40 observations and the second has 60. Let \(A\) be the standardized sum for the first group and \(B\) the standardized sum for the second. Under the same null model, both have variance 1, but no observation appears in both sums. Therefore

\operatorname{Cov}(A,B)=0.

Because the two sums are jointly normal, zero covariance implies that \(A\) and \(B\) are independent. For instance, if their observed standardized sums were \(2.214\) and \(-1.549\), respectively, their statistics would be \(A=2.214/\sqrt{40}\approx0.350\) and \(B=-1.549/\sqrt{60}\approx-0.200\). Those values do not alter the model-based conclusion that the statistics are independent. This is a different design from cumulative monitoring: the later group does not retain the earlier observations.

Assumptions, Diagnostics, and Limits

Before using the covariance formula, check the design and model that produced it. In particular, verify that the later analysis truly includes the earlier observations, that observations are independent across participants, and that the standardized observations have the same variance. The exact joint-normal argument also relies on normal observations under \(H_0\); with nonnormal data, a normal approximation may be reasonable in some settings, but it is not the exact result derived here.

The calculation may not apply as written when the analysis uses a different statistic, estimates the variance, has repeated measurements within participants, uses unequal variances, or changes the sample-size or allocation rule in response to accumulating results. In those settings, shared observations still call for careful examination, but neither the covariance formula nor the conclusion of dependence should be assumed without a derivation under the relevant model. A useful diagnostic is to write each statistic in terms of its observations and identify which terms are shared, then calculate the covariance using the assumptions actually justified.

The comparison with disjoint samples clarifies the main structural difference. Separate independent samples can produce independent statistics under suitable conditions; nested cumulative samples produce a positive covariance in the model studied here. This tutorial establishes that relationship, not the size of any overall error rate. The actual effect of dependence on repeated-testing error requires a separate calculation of the joint rejection event.

Key takeaway: In the cumulative normal mean model with known variance, the shared-sum structure gives \(\operatorname{Corr}(Z_j,Z_k)=\sqrt{n_j/n_k}\) for \(j<k\). The positive covariance establishes dependence in this model; sharing data alone is not a general proof.

Check Your Understanding

Answer each question, then compare with the brief solutions.

Under the cumulative model, what is the correlation between looks at \(n_1=25\) and \(n_2=100\)?
Why does the covariance between two cumulative statistics include contributions from only the observations present at both looks?
At \(n_1=49\) and \(n_2=100\), what fraction of the second statistic's representation is the coefficient multiplying \(Z_1\)? What does that coefficient represent?
Why does zero covariance imply independence in the model used here, but not necessarily in every statistical model?
Give two assumptions or design features that would need checking before using the stated covariance formula in an actual trial.

Solutions. (1) \(\sqrt{25/100}=0.500\). (2) All observations unique to the second look are independent of the first-look observations, so their cross-covariances with the first statistic are zero; only the shared observations contribute. (3) The coefficient is \(\sqrt{49/100}=0.700\). It is the weight with which the first-look statistic contributes to the second-look statistic. (4) The statistics are jointly normal in this model, and jointly normal variables with zero covariance are independent. Outside joint normality, zero covariance alone does not generally imply independence. (5) Examples include checking that later looks include earlier observations, that observations are independent across participants, that their variances are equal, and that the assumed normal model or an appropriate approximation is justified.