How Dependence Changes Error Inflation

Why Dependence Matters

When a trial is tested repeatedly, the probability of a false rejection at any look depends not only on each test’s significance level but also on how the rejection events overlap. If two tests use much of the same data, their results tend to move together. A false rejection at one look is then more likely to coincide with a false rejection at another, so the tests create fewer distinct opportunities for a false positive than independent tests with the same per-look error rate.

That reduction is not the same as eliminating inflation. If either look can trigger a rejection, the relevant probability is the chance of at least one rejection under the trial-wise null hypothesis \(H_0\). Even with positive dependence, that probability can remain larger than the per-look significance level \(\alpha\).

Dependence in Cumulative Tests

A standard model makes the source of dependence explicit. Suppose \(X_1,\ldots,X_{n_2}\) are independent observations from \(N(\mu,\sigma^2)\), with known \(\sigma\), and consider testing \(H_0:\mu=\mu_0\). At two planned looks, \(n_1<n_2\), define the standardized statistics

where \(\bar{x}_j\) is the mean of the first \(n_j\) observations. Under \(H_0\), each statistic has a standard normal distribution. The second statistic reuses the observations from the first look; the two statistics are therefore not independent.

To derive their correlation, write \(Z_j=S_j/\sqrt{n_j}\), where \(S_j=\sum_{i=1}^{n_j}(X_i-\mu_0)/\sigma\). Under \(H_0\), each summand has mean zero and variance one. The first \(n_1\) summands appear in both sums, while the remaining summands in \(S_2\) are independent of \(S_1\). Thus \(\operatorname{Cov}(S_1,S_2)=n_1\), and

This correlation is positive, and it is larger when the looks are closer in sample size. The formula relies on independent observations, the normal model with known variance, the null hypothesis, and a cumulative design in which the earlier observations are retained. In other models, the exact correlation formula may differ.

How Overlap Changes the Any-Look Probability

For two rejection events, the probability of at least one rejection follows directly from inclusion–exclusion:

The intersection is subtracted because outcomes in which both tests reject would otherwise be counted twice. If both marginal error probabilities are \(\alpha\), then the any-look probability is \(2\alpha-P(A_1\cap A_2\mid H_0)\). Under independence, the intersection equals \(\alpha^2\). Under positive dependence, it is larger than \(\alpha^2\), so the union probability is smaller than the independent benchmark. Unless the events overlap completely, however, the union can still exceed \(\alpha\).

Positive Dependence: What It Does and Does Not Guarantee

For two tests with equal marginal error probabilities, the comparison with independence is exact once the joint rejection probability is known. If positive dependence means \(P(A_1\cap A_2\mid H_0)>\alpha^2\), then

The conclusion is that the dependent tests have a lower any-look error probability than independent tests with the same marginal error probabilities. It does not say the error probability equals \(\alpha\), or that it is necessarily close to \(\alpha\).

For more than two looks, the same principle is that overlapping rejection events leave fewer distinct outcomes in their union. But knowing only that every pair of tests has positive correlation does not, in an arbitrary model, determine the probability of the union. The joint distribution of all the rejection events matters. In the cumulative normal setting, the statistics are jointly normal with nonnegative correlations, and their rejection patterns reflect the shared data. The exact familywise probability still depends on the schedule, the boundaries, and the joint distribution.

It is also important to distinguish correlation between test statistics from positive dependence between rejection events. A positive correlation describes how the numerical statistics co-vary. Rejection events are threshold-based, and their overlap depends on the thresholds and on the distribution’s shape. The bivariate normal example supports the expected comparison for its specific assumptions; it is not a universal conversion from a correlation value to an error probability.

Comparison, Diagnostics, and Limitations

A practical diagnostic question is whether later tests reuse earlier observations. If they do, treating their statistics as independent is usually inconsistent with the data structure. Also check that the event being discussed is the union of all rejection events, rather than a rejection at one particular look. The introductory material on per-look versus overall significance distinguishes these error rates; dependence explains why the overall rate cannot be obtained by simply treating each look as an independent new trial.

The independence calculation remains useful as a reference point, not as a default answer for cumulative tests. Conversely, positive dependence should not be used as a justification for ignoring repeated testing. The exact amount of inflation must be assessed for the design and analysis being used. These comparisons do not specify adjusted stopping boundaries or account for departures from the assumed model.

A Common Mistake

Wrong reasoning: “The interim statistics are positively correlated, so repeated testing does not inflate the false-positive rate.” Positive dependence generally increases overlap, but it does not make all rejection events identical. In the two-look example, the any-look probability remains about \(9.08\%\), not \(5\%\).

Correction: Calculate or otherwise justify the joint rejection probability under the null, then use the union probability. Correlation gives information about dependence, but it is not itself the familywise error rate.