Multiplicity from Repeated Predictor Searches

Why Repeated Predictor Searches Matter

A regression search may test several candidate predictors and retain those that meet an entry rule. Each test can have a small false-positive risk on its own, yet searching many predictors creates multiple opportunities for a false entry. The key quantity is therefore not only the error probability for one test, but also the chance of at least one false rejection across the search.

For a simple starting point, imagine a fixed family of \(m\) candidate-predictor tests, each with a valid p-value under its own null hypothesis. A p-value is valid if, whenever its null hypothesis is true, \(P(p\leq u)\leq u\) for every \(u\) from 0 to 1. This condition is also called super-uniformity. It ensures that testing at threshold \(\alpha\) has Type I error probability at most \(\alpha\), not necessarily exactly \(\alpha\).

The family must be stated clearly. It might consist of all candidate predictors tested once, or it might include the tests considered at several stages of a selection path. If there are true and false null hypotheses, the FWER concerns false rejections among the true nulls, not the rejection of a genuinely nonzero effect. The nominal level \(\alpha\) describes a single test; it does not, by itself, promise that the FWER for a search is at most \(\alpha\).

The Independent-Tests Benchmark

Suppose all \(m\) null hypotheses are true, and each test has false-rejection probability exactly \(\alpha\). If the tests are independent, then their rejection events are independent. The probability of no false rejections is the probability that every test does not reject, so it equals \((1-\alpha)^m\). Taking the complement gives the chance of at least one false rejection.

This formula is exact when the tests are independent and each test has Type I error probability exactly \(\alpha\). An exact continuous p-value under its null hypothesis is uniformly distributed on 0 to 1, so it falls at or below \(\alpha\) with probability exactly \(\alpha\). A merely valid p-value need not be uniform: it is super-uniform and has probability at most \(\alpha\) of falling at or below \(\alpha\). If the tests are independent but conservative, the formula with \(\alpha\) gives an upper bound rather than an exact FWER.

The result grows with the number of tests because each additional test creates another opportunity for a false rejection. For example, with twenty independent exact tests at level \(0.05\), the FWER is \(1-0.95^{20}\), approximately \(0.6415\). The individual threshold remains \(0.05\), but the chance of at least one false positive is about \(64.15\%\).

What Can Be Said Without Independence?

Predictor tests in regression are often dependent: they may use the same response values, and candidate predictors may be correlated. The independent-tests formula should not be applied as though independence were automatic. A general bound is available without an independence assumption.

Let \(A_j\) be the event that test \(j\) falsely rejects its null hypothesis. The event of at least one false rejection is the union \(A_1\cup\cdots\cup A_m\). The union bound says that the probability of a union is no greater than the sum of its event probabilities. Since each valid level-\(\alpha\) test has \(P(A_j)\leq\alpha\) when its null is true, it follows that

If the family contains fewer than \(m\) true null hypotheses, apply the same reasoning to the tests of those true nulls. This bound can be conservative, but it remains valid under dependence when each relevant test is valid. If \(m\alpha\) exceeds 1, the probability is still at most 1; the bound is then not informative.

A Repeated-Testing Simulation

A simulation can make the distinction between a nominal p-value and a search-wide error rate concrete. For a clean benchmark, generate independent uniform p-values under a setting where all tested null hypotheses are true. Each simulated repetition represents a new hypothetical search, and a false rejection occurs if at least one of its p-values falls below the chosen threshold. This illustrates repeated testing; it does not reproduce every dependence pattern or adaptive feature of a particular regression algorithm.

In an actual predictor search, p-values may be dependent, and a stepwise procedure may revisit the candidate set at successive stages. The simulation above therefore serves as a transparent reference case, not as a universal estimate for stepwise selection. To assess a particular procedure by simulation, reproduce its search rules and data-generating conditions in each repetition, then record whether it makes at least one false entry.

Limiting the Search-Wide Risk

One simple way to control FWER is the Bonferroni rule: for a fixed family of \(m\) tests, use a per-test threshold of \(\alpha/m\). Under the validity condition for each test, the union bound then gives an FWER no greater than \(\alpha\), without requiring independence. This method can be conservative, especially when test results are strongly dependent, but its guarantee is straightforward.

The phrase specified family matters. In a search that performs tests repeatedly, counting only the predictors in the data set may omit later testing opportunities. The relevant family may include all tests that the procedure could perform, and the validity of the p-values must be considered in light of how those tests are chosen. A simple adjustment applied to nominal p-values does not automatically repair invalid p-values or account for every adaptive feature of a search.

Common Mistakes and Practical Meaning

• Equating a nominal level with search-wide protection. A test at level \(0.05\) controls its own Type I error at no more than \(0.05\); it does not generally ensure an FWER of \(0.05\) across many tests.
• Using the independence formula without checking independence. Predictor tests may be dependent. Use \(1-(1-\alpha)^m\) as an exact result only under its stated conditions.
• Assuming every valid p-value is exactly uniform. Validity requires \(P(p\leq u)\leq u\). Exact continuous p-values are uniform; conservative valid p-values can have a smaller rejection probability.
• Confusing a false-rejection probability with the probability a null hypothesis is true. FWER is a probability of an event under a testing setup; it is not the probability that a particular null hypothesis is true after observing a p-value.
• Ignoring how many tests the procedure actually performs. If selection revisits candidates or tests at multiple stages, the family and any adjustment must reflect the search being considered.

Multiplicity matters because a predictor that enters after a broad search has passed more opportunities to look compelling than a predictor tested once in a prespecified analysis. A small nominal p-value can still be evidence against a null hypothesis, but interpreting it requires knowing how the search was conducted and what family of tests is relevant. Describing the search and its multiplicity treatment makes the strength of the evidence easier to assess.