Look Timing and Dependence Between Tests

Why the Timing of Looks Matters

Two monitoring schedules can have the same number of interim analyses and the same per-look threshold, yet the statistics at those looks may have very different dependence. If looks occur close together, most observations contributing to the later statistic also contributed to the earlier one. The results therefore tend to move together. If there is a long gap between looks, more new data enter, and the later statistic can differ more substantially from the earlier one.

This distinction matters for repeated rejection. A rejection at one look does not imply that the next look will reject, but the probability of a second rejection depends on how much information the looks share. Dependence affects the pattern of rejection across looks, even when each statistic has the same null distribution and each individual test uses the same threshold.

A Model for Cumulative Tests

To make the timing effect precise, suppose observations arrive in independent increments. Under the null hypothesis, let each standardized observation $Y_i$ have distribution $N(0,1)$, independently of the others. At sample size $n$, define the cumulative standardized statistic

S_n=\sum_{i=1}^{n}Y_i,\qquad Z_n=\frac{S_n}{\sqrt{n}}.

Then $Z_n\sim N(0,1)$ under the null. This is an idealized known-variance model: observations are independent, their null means are zero, their variances are one, and the statistic at each look uses all observations collected so far. In applications, a normal approximation or a suitable standardized score statistic may play the same role. The correlation formula below applies to cumulative statistics with independent increments and information proportional to sample size.

Definition: For two cumulative looks at sample sizes $n_1<n_2$, the dependence between their standardized statistics is summarized by $$ \operatorname{Corr}(Z_{n_1},Z_{n_2})=\sqrt{\frac{n_1}{n_2}}. $$ The ratio $n_1/n_2$ is the fraction of the later look’s information already present at the earlier look.

The correlation is positive because the earlier observations are part of both statistics. It is close to one when $n_2$ is only slightly larger than $n_1$. It decreases as the later look moves farther away in information time. It does not become zero at any finite $n_2$ in this cumulative model, because the later statistic still contains the earlier observations.

Why the Correlation Has This Form

The earlier sum $S_{n_1}$ is included in $S_{n_2}$. Write $S_{n_2}=S_{n_1}+(S_{n_2}-S_{n_1})$, where the added observations are independent of $S_{n_1}$. Under the stated model, $\operatorname{Var}(S_{n_1})=n_1$ and $\operatorname{Var}(S_{n_2})=n_2$. Therefore,

\operatorname{Cov}(Z_{n_1},Z_{n_2}) =\operatorname{Cov}\left(\frac{S_{n_1}}{\sqrt{n_1}}, \frac{S_{n_1}+(S_{n_2}-S_{n_1})}{\sqrt{n_2}}\right) =\frac{n_1}{\sqrt{n_1n_2}} =\sqrt{\frac{n_1}{n_2}}.

Both standardized statistics have variance one, so their covariance is also their correlation. This derivation shows that dependence is not an extra assumption about researchers’ decisions: it follows from the shared observations in the cumulative statistics.

Worked Example: Comparing Close and Widely Spaced Looks

Consider a hypothetical monitoring plan with a look at $n_1=100$. Compare a second look at $n_2=110$ with one at $n_2=400$. Assume the independent-increment null model above.

For the close pair, $$ \operatorname{Corr}(Z_{100},Z_{110})=\sqrt{\frac{100}{110}}=\sqrt{0.9091}\approx 0.9535. $$ The two statistics are very strongly positively correlated: only 10 new observations enter before the second look.

For the widely spaced pair, $$ \operatorname{Corr}(Z_{100},Z_{400})=\sqrt{\frac{100}{400}}=\sqrt{0.25}=0.5000. $$ The later statistic still uses the first 100 observations, but those observations make up a smaller fraction of its total information. The correlation is therefore lower than for the close pair.

This comparison concerns dependence, not the individual null distribution: each $Z_n$ is still standard normal under the model. Also, a correlation of 0.5000 does not mean that half of the time the statistics agree; it describes their linear association across repeated realizations under the assumed model.

Conditional Behavior Across Looks

Correlation gives an overall measure of how two statistics move together. A conditional distribution gives a more direct answer to a monitoring question: if the earlier statistic has a particular observed value, what values are plausible at the later look? For $n_1<n_2$, split the later statistic into the contribution from the earlier data and the contribution from new data:

Z_{n_2} =\sqrt{\frac{n_1}{n_2}}Z_{n_1} +\frac{S_{n_2}-S_{n_1}}{\sqrt{n_2}}.

The second term is independent of $Z_{n_1}$. Its mean is zero and its variance is $(n_2-n_1)/n_2$. Thus, under the null model, conditional on $Z_{n_1}=z$,

Z_{n_2}\mid Z_{n_1}=z \sim N\left(\sqrt{\frac{n_1}{n_2}}z,\;1-\frac{n_1}{n_2}\right).

The conditional mean preserves part of the earlier signal, with the amount preserved determined by the correlation. The conditional variance reflects the new data. A close look has a large conditional mean contribution from the earlier statistic and relatively little new-data variation. A more distant look has more new data and, accordingly, more opportunity for the statistic to shift.

Worked Example: The Next Statistic After an Early Rejection

Suppose a hypothetical study uses the one-sided rejection rule $Z_n>1.96$ at each look. This is a per-look upper-tail threshold; the calculation here describes conditional behavior and does not itself adjust the threshold for repeated testing. At $n=100$, suppose the observed statistic is $Z_{100}=2.20$. Find the conditional mean and standard deviation of $Z_{110}$.

The correlation is $\sqrt{100/110}=\sqrt{10/11}\approx 0.9534626$. Therefore, $$ E(Z_{110}\mid Z_{100}=2.20) =2.20\sqrt{\frac{10}{11}} \approx 2.097618, $$ and $$ SD(Z_{110}\mid Z_{100}=2.20) =\sqrt{1-\frac{100}{110}} =\sqrt{\frac{1}{11}} \approx 0.301511. $$

The conditional mean remains close to 2.20, and the conditional standard deviation is small. This is the numerical effect of a very close look: the later statistic contains nearly all of the information behind the earlier result.

Worked Example: Comparing Conditional Rejection Probabilities

Continue with the hypothetical upper-tail threshold $1.96$ and the observed value $Z_{100}=2.20$, but compare a second look at $n=110$ with one at $n=400$. For a normal variable with conditional mean $m$ and standard deviation $s$, the conditional probability of exceeding 1.96 is $1-\Phi((1.96-m)/s)$, where $\Phi$ is the standard normal cumulative distribution function.

For the close look at $n=110$, the conditional mean and standard deviation are $2.0976177$ and $0.3015113$, using unrounded values from the formulas. The standardized threshold is $$ \frac{1.96-2.20\sqrt{10/11}}{\sqrt{1/11}} \approx -0.4564. $$ Hence, $$ P(Z_{110}>1.96\mid Z_{100}=2.20) =1-\Phi(-0.4564) =\Phi(0.4564) \approx 0.6760. $$

For the distant look at $n=400$, the conditional mean is $2.20\sqrt{100/400}=1.10$, and the conditional standard deviation is $\sqrt{1-100/400}=\sqrt{0.75}\approx0.8660$. The standardized threshold is $$ \frac{1.96-1.10}{\sqrt{0.75}}\approx 0.9930, $$ so $$ P(Z_{400}>1.96\mid Z_{100}=2.20) =1-\Phi(0.9930)\approx 0.1603. $$

Given this particular early statistic, a rejection at the close look is much more likely than at the distant look. For comparison, under independence the conditional probability would equal the marginal per-look upper-tail probability $1-\Phi(1.96)\approx0.0250$. Neither conditional probability says that a later rejection is certain or that the null hypothesis has changed; these are probabilities under the specified null model, conditional on the stated earlier statistic.

Implications for Repeated Rejection

When looks are close, a statistic that crosses a threshold is likely to remain near that threshold at the next look, although it can still move back across it. A sequence of nearby looks can therefore produce clusters of similar results. With wider spacing, new observations contribute a larger share of the later statistic, so its value can be less predictable from the earlier one. These are tendencies under the model, not guarantees for any one trial.

Timing affects the probability of paths through the sequence of tests—for example, rejection at one look followed by non-rejection at another. It also affects the probability of at least one rejection across the schedule. The individual test’s null rejection probability alone does not determine that overall probability; the joint distribution of the statistics matters. This is why treating cumulative looks as independent can be misleading, particularly when looks are close together.

The main alternatives are useful as reference points, but neither should be mistaken for the actual cumulative model. Under independent tests, each look contributes a fresh statistic with no shared observations, so conditional rejection probability is unchanged by earlier results. Under perfect dependence, every look produces the same standardized statistic, so a rejection at one look is repeated at every look. Cumulative tests with distinct finite sample sizes generally lie between these extremes, with their dependence determined by the information overlap.

Spacing is not the only determinant of dependence in real analyses. The formula $\sqrt{n_1/n_2}$ relies on the independent-increment model and information proportional to sample size. If observations are dependent, the test statistic changes its definition across looks, or information accrues unevenly, the correlation may differ. In those settings, use the appropriate joint model rather than applying the sample-size formula mechanically.

Diagnostics and Common Pitfalls

Check the information scale. The sample-size ratio is suitable only when sample size is proportional to information and the cumulative statistic has independent increments. If that is not justified, the formula may not describe the actual dependence.
Distinguish marginal from conditional probabilities. Every $Z_n$ can have the same standard normal null distribution while $Z_{n_1}$ and $Z_{n_2}$ are strongly dependent. Equal marginal distributions do not imply independence.
Do not equate correlation with a probability of repeated rejection. A correlation such as 0.9535 is not the probability of rejecting twice. The latter also depends on the rejection threshold and on the joint distribution.
Do not treat a long gap as independence. A distant cumulative look still includes earlier observations. Its correlation is lower in this model, but remains positive for finite sample sizes.
Do not infer that spacing alone controls overall error. The number of looks, the rejection rule, the boundary, and the dependence all matter. Timing comparisons describe one part of the repeated-testing problem, not a complete error-control procedure.

A practical diagnostic is to compare the planned information at each look and calculate the implied correlations under the working model. If observed or simulated statistics show markedly different dependence, investigate the data structure, information scale, and statistic construction before interpreting the schedule using this model.

Key takeaway: Under cumulative independent-increment testing, the correlation between two standardized looks is the square root of the earlier-to-later information ratio. Close looks share more information, so a rejection is more likely to persist immediately; wider spacing allows more new data to alter the later statistic.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions below.

Under the cumulative model, calculate the correlation between looks at $n_1=80$ and $n_2=100$.
For looks at $n_1=60$ and $n_2=240$, what fraction of the later look’s information was already present at the earlier look, and what is the correlation?
If $Z_{80}=1.50$, find the conditional mean and standard deviation of $Z_{100}$ under the null model.
Why is it incorrect to say that a correlation of 0.50 means the second test has a 50% chance of repeating the first test’s rejection?
Two schedules have the same number of looks and the same per-look threshold. One has closely spaced cumulative looks and the other widely spaced looks. Which schedule has greater dependence between consecutive statistics under this model, and why?

Solutions. (1) $\sqrt{80/100}=\sqrt{0.8}\approx0.8944$. (2) The earlier information fraction is $60/240=0.25$, and the correlation is $\sqrt{0.25}=0.5000$. (3) The conditional mean is $1.50\sqrt{80/100}\approx1.342$; the conditional standard deviation is $\sqrt{1-80/100}=\sqrt{0.2}\approx0.4472$. (4) Correlation describes association between statistic values, not the conditional probability of a threshold crossing. That probability also depends on the observed statistic and the threshold. (5) The closely spaced schedule has greater dependence because a larger fraction of the later statistic’s information is shared with the preceding look.