Why Screening Many Predictors Changes the Evidence
Univariate screening tests each predictor against an outcome separately. This can be useful when a data set contains many candidate predictors, but it creates a multiple-testing problem: even if every predictor is unrelated to the outcome in the population, some tests may produce small p-values by chance. The more tests performed, the more opportunities there are to see an apparently compelling result.
A small unadjusted p-value is evidence against one null hypothesis under the conditions of that test. It is not, by itself, evidence that the predictor will improve a model, that its association is causal, or that the result will replicate in new data. When many predictors have been examined, the number of tests must be part of the interpretation.
The family should be defined by the screening question, not retrospectively narrowed to whichever results look interesting. If 60 predictors were tested but only the 8 smallest p-values are reported, the 60 tests still matter when making a claim based on that search. If several outcomes or several alternative tests were also searched, the relevant family may be broader than a single list of predictors.
False Rejections: Expected Count and Family-Wise Risk
Suppose \(m_0\) null hypotheses in a family are true, and each corresponding test has a valid p-value. Testing each at level \(\alpha\) means that the probability of rejecting any particular true null is at most \(\alpha\). It need not equal \(\alpha\): a valid but conservative test can have a lower rejection probability.
This result follows by writing \(V\) as a sum of indicator variables. Let \(I_j=1\) if true null \(j\) is rejected and \(I_j=0\) otherwise. Then \(V=\sum_{j\in H_0}I_j\). By linearity of expectation, \(E[V]=\sum_{j\in H_0}E[I_j]=\sum_{j\in H_0}P(\text{reject }j)\). Validity gives \(P(\text{reject }j)\leq\alpha\) for each true null, so summing these inequalities gives \(E[V]\leq m_0\alpha\). No independence assumption is used in this derivation.
An expected count is not the probability of at least one false rejection. A separate, simple bound for that probability comes from the union bound: $$ P(V\geq 1)\leq \sum_{j\in H_0}P(\text{reject }j)\leq m_0\alpha. $$ This bound also does not require independence, though it can be loose. If the true-null rejection decisions are independent and each has probability exactly \(\alpha\), then \(P(V\geq1)=1-(1-\alpha)^{m_0}\). With independent valid tests whose individual rejection probabilities are only known to be at most \(\alpha\), this expression is an upper bound, not necessarily an exact probability.
Worked Example: Interpreting 40 Unadjusted Tests
A hypothetical analyst tests 40 candidate predictors against one outcome, using valid p-values and an unadjusted level of \(\alpha=0.05\) for each. Assume all 40 null hypotheses are true. Find an upper bound for the expected number of false rejections. Then suppose the test decisions are independent and find an upper bound for the probability of at least one false rejection.
Method and assumptions. Use the expected-count bound and, for the second calculation, independence of the decisions. Validity alone does not imply that a test rejects with probability exactly 0.05.
There are \(m_0=40\) true nulls, so $$ E[V]\leq m_0\alpha=40(0.05)=2. $$ Thus the expected number is at most 2. It equals 2 only if every true-null test has type I error exactly 0.05.
Under independence, each test has probability at least \(1-0.05=0.95\) not to reject. Therefore, $$ P(V=0)\geq 0.95^{40}\approx 0.1285, $$ and $$ P(V\geq1)\leq 1-0.95^{40}\approx 0.8715. $$ The probability is at most 0.8715; it equals that value if every test rejects its true null with probability exactly 0.05. The calculations are consistent: the expected count bound is 2, while the probability bound describes whether the count is positive, not its average size.
Check. Since a probability cannot exceed 1, the computed bound of approximately 0.8715 is within the allowable range. The common mistake is to state an exact expected count or exact probability using only the fact that each test is valid at level 0.05.
Bonferroni and Holm Adjustments
A family-wise procedure controls the probability of one or more false rejections across the family. Bonferroni achieves this by testing each hypothesis at a smaller level. To target a family-wise error rate no greater than \(\alpha\), compare each valid p-value with \(\alpha/m\), where \(m\) is the total number of hypotheses in the family. This works without assuming independence: the union bound gives \(P(\text{at least one false rejection})\leq m(\alpha/m)=\alpha\). If fewer than \(m\) nulls are true, the bound remains valid.
Equivalently, a Bonferroni-adjusted p-value for hypothesis \(j\) is \(\min(1,mp_j)\). Compare that adjusted value with the family-wise target \(\alpha\). The adjustment does not change the original test statistic; it changes the threshold used to account for the search across the family.
Worked Example: Applying a Bonferroni Threshold
A hypothetical screen tests 24 predictors against one outcome. The family-wise target is \(\alpha=0.05\). The three smallest p-values are 0.0012, 0.0020, and 0.0022; every other p-value is greater than 0.0022. Which hypotheses pass the Bonferroni procedure?
Method and assumptions. Use the Bonferroni threshold for all \(m=24\) tests. Assume the reported p-values are valid for their null hypotheses; no independence assumption is needed.
First calculate the per-test threshold: $$ \frac{\alpha}{m}=\frac{0.05}{24}=0.0020833\ldots $$ Now compare the ordered p-values with this threshold. The first value satisfies \(0.0012<0.0020833\), so it passes. The second satisfies \(0.0020<0.0020833\), so it also passes. The third does not pass because \(0.0022>0.0020833\). Since all remaining p-values are larger still, they do not pass either.
As a check using adjusted p-values, the first two are \(24(0.0012)=0.0288\) and \(24(0.0020)=0.0480\), both at most 0.05. The third is \(24(0.0022)=0.0528\), which is greater than 0.05. Therefore, Bonferroni rejects the first two null hypotheses and no others.
Common mistake avoided. Comparing the smallest p-values with 0.05 would ignore the 24-test search. Passing the adjusted screen is evidence against the relevant nulls under the procedure; it does not establish that the predictors are useful in a final predictive model.
Holm’s step-down procedure also controls FWER without requiring independent tests, and is never less powerful than applying the same Bonferroni threshold to every p-value. Order the \(m\) p-values from smallest to largest. Compare the \(k\)th ordered value with \(\alpha/(m-k+1)\), starting at \(k=1\). Continue while each comparison passes. Stop at the first failure and reject no hypotheses at that rank or later.
The thresholds become less stringent as the procedure proceeds. However, Holm is not a rule to test each p-value against its own threshold independently: the stopping rule matters. Once a comparison fails, a later p-value cannot be rejected by continuing down the list.
False Discovery Rate: A Different Goal
When screening many predictors, requiring strong control of the chance of even one false rejection can be very conservative. A different target is the false discovery rate (FDR), the expected proportion of rejections that are false. Define \(R\) as the total number of rejections and \(V\) as the number of false rejections; the false discovery proportion is \(V/R\) when \(R>0\), and is defined as 0 when \(R=0\). The FDR is \(E[V/R]\) under that convention.
The Benjamini–Hochberg procedure controls FDR at target \(q\) under independence of the p-values, and under certain forms of positive dependence. It is not a general guarantee under arbitrary dependence. Order the p-values \(p_{(1)}\leq\cdots\leq p_{(m)}\), and find the largest rank \(k\) such that \(p_{(k)}\leq kq/m\). Reject the hypotheses corresponding to the first \(k\) ordered p-values. If no rank meets its threshold, reject none.
Worked Example: Benjamini–Hochberg for a Screening List
A hypothetical analyst tests eight predictors. The ordered p-values are 0.001, 0.008, 0.012, 0.018, 0.031, 0.044, 0.120, and 0.400. The target FDR is \(q=0.10\). Assume independent p-values. Apply the Benjamini–Hochberg procedure.
Method and assumptions. Compare each ordered p-value with \(kq/m\), where \(m=8\). The stated independence assumption is one condition under which this procedure controls FDR at the target.
The ordered thresholds for ranks 1 through 8 are \(0.0125, 0.0250, 0.0375, 0.0500, 0.0625, 0.0750, 0.0875,\) and \(0.1000\), respectively. At rank 6, \(0.044\leq0.075\), so the condition passes. At rank 7, \(0.120>0.0875\), so it fails; rank 8 also fails because \(0.400>0.1000\). The largest passing rank is \(k=6\).
The procedure rejects the first six hypotheses and not the final two. This is a decision rule targeting the expected false proportion among rejections, not a guarantee that exactly 10% of these six rejections are false. The common mistake is to interpret \(q=0.10\) as a 10% probability that any particular selected predictor is false.
Choosing and Reporting an Adjustment
The adjustment should match the purpose of the analysis. If the claim requires protection against any false rejection in a defined family, use a family-wise procedure such as Bonferroni or Holm. If the purpose is to produce a broader exploratory list while controlling the expected false proportion, an FDR procedure may be more appropriate, provided its dependence conditions are defensible.
| Approach | Primary quantity controlled | Independence required? |
|---|---|---|
| Unadjusted tests | Per-test type I error, not family-wide error | No, for the per-test interpretation |
| Bonferroni | Family-wise error rate | No |
| Holm | Family-wise error rate | No |
| Benjamini–Hochberg | False discovery rate | Independence or suitable positive dependence for the standard guarantee |
Predictor tests are often dependent because predictors can be correlated, and because tests share the same outcome and observations. Dependence does not invalidate Bonferroni or Holm control, but it affects which guarantees can be claimed for an FDR procedure. Whatever method is used, report the number and definition of tests in the family, the adjustment method, the target level, and the resulting adjusted evidence. Do not silently omit unpromising tests from the family after seeing their p-values.
Check Your Understanding
For each problem, use the stated assumptions carefully; a bound is not an exact value unless the problem gives exact type I error probabilities.
- A screen has 30 true null hypotheses, each tested with a valid p-value at level 0.02. Find an upper bound for the expected number of false rejections. (Answer: At most \(30(0.02)=0.6000\).)
- Twelve true-null test decisions are independent, and each test rejects with probability exactly 0.05. Find the probability of at least one false rejection. (Answer: \(1-0.95^{12}\approx0.4596\).)
- A family contains 18 tests and uses Bonferroni with family-wise target 0.05. The three smallest p-values are 0.0011, 0.0025, and 0.0030; all others are larger. How many hypotheses are rejected? (Answer: Threshold \(0.05/18\approx0.0027778\); reject the first two.)
- Four ordered p-values are 0.006, 0.014, 0.021, and 0.30. Apply Holm at family-wise target 0.05. (Answer: Reject the first three; the fourth fails its threshold of 0.05.)
- Six ordered p-values are 0.004, 0.012, 0.020, 0.035, 0.080, and 0.20. Apply Benjamini–Hochberg at \(q=0.10\), assuming independence. (Answer: The largest passing rank is 5; reject the first five hypotheses.)