Why Adjusted Thresholds Matter in Covariate Screening
A covariate search may test whether age, body size, renal function, genotype, or a co-medication is associated with a pharmacokinetic parameter. When several hypotheses are tested, comparing every p-value with the same nominal significance level can produce more false-positive findings than intended. The issue is not resolved by interpreting each test in isolation: the relevant error rate depends on the full set of decisions.
This tutorial focuses on threshold procedures for a defined family of covariate hypotheses. The tests might come from a prespecified screening analysis, or from a set of candidate tests whose inferential properties are otherwise justified. Multiplicity adjustments address the chance of erroneous rejections across that family; they do not repair invalid individual p-values, model misspecification, or a search strategy that changes which hypotheses are tested based on the same data.
FWER control is stringent: it limits the probability of any false positive in the family. FDR control instead limits the long-run expected fraction of false findings among rejected hypotheses. Neither statement means that a particular rejected covariate effect has a specified probability of being false.
Bonferroni and Holm: Controlling the Family-Wise Error Rate
Suppose \(m\) hypotheses are tested, and each null p-value is valid: when its null hypothesis is true, the p-value is at least as likely to exceed any given threshold as a Uniform(0,1) random variable. The tests need not be independent for the Bonferroni or Holm guarantees described here. The family should be defined before interpreting the results, and the target FWER is denoted by \(\alpha\).
The Bonferroni guarantee follows from the union bound. Let \(V\) be the number of false rejections, and let \(I_0\) be the set of true null hypotheses. A false rejection occurs only if at least one true-null p-value falls below \(\alpha/m\). Therefore:
The argument uses valid marginal p-values and does not require independence. It may be conservative, particularly when many tests are performed or tests are strongly correlated.
Worked Example: Bonferroni Screening Threshold
In a hypothetical analysis, investigators prespecify five tests of covariate effects on clearance, with target FWER \(\alpha=0.05\). The ordered p-values are \(0.003,\ 0.011,\ 0.018,\ 0.041,\ 0.200\). Assume each test supplies a valid p-value under its null.
Bonferroni uses the same cutoff for every test:
Only \(0.003\leq0.010\), so Bonferroni rejects the hypothesis associated with the smallest p-value and does not reject the other four. Under the stated assumptions, the probability of at least one false rejection in this five-test family is at most 0.05. This is a family-level guarantee, not a 5% probability that the particular rejected effect is false.
Holm’s step-down procedure also controls FWER under arbitrary dependence among valid p-values and is never less powerful than Bonferroni for the same family and \(\alpha\). Order the p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\). Starting with the smallest, compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Continue while each comparison passes; at the first failure, stop and do not reject that hypothesis or any with a larger p-value.
Worked Example: Holm’s Step-Down Decisions
Use the same hypothetical five p-values and \(\alpha=0.05\). The ordered comparisons are:
The first two comparisons pass and the third fails. Holm therefore rejects the first two hypotheses only. The procedure stops at the first failure; it does not continue testing the remaining p-values against their later, larger thresholds. For this example, Holm identifies one additional rejection compared with Bonferroni while retaining the FWER guarantee.
A justification for Holm’s guarantee also uses the union bound, but applies the thresholds in order. If there are \(t\) true null hypotheses, then the first true null encountered in the ordered list has rank no greater than \(m-t+1\). For that true null to be rejected, its p-value must pass a threshold no greater than \(\alpha/t\). The probability that any true-null p-value is at most \(\alpha/t\) is bounded by \(t(\alpha/t)=\alpha\). Thus the probability of any false rejection is at most \(\alpha\), without an independence requirement.
Benjamini–Hochberg and Benjamini–Yekutieli: Controlling the FDR
When a screening analysis is intended to produce a set of candidates for further evaluation, controlling the expected fraction of false discoveries may be more relevant than controlling the chance of even one false rejection. The Benjamini–Hochberg procedure (BH) targets FDR. Its standard guarantee holds when the p-values are independent, and also under certain positive dependence conditions, commonly expressed as positive regression dependence on the subset of true nulls (PRDS). It is not a general guarantee for arbitrary dependence.
The threshold increases with rank. A small p-value can pass even if it is larger than the Bonferroni cutoff, because BH allows a larger cutoff for later ranks. The rule is based on the largest passing rank: once that rank is found, all hypotheses up to it are rejected, including any earlier p-values that pass their own rank thresholds.
Worked Example: BH and BY With Correlated Covariate Tests
Consider four hypothetical tests of covariate effects on apparent clearance, with ordered p-values \(0.004,\ 0.012,\ 0.018,\ 0.040\), and target \(q=0.05\). Suppose the tests may be dependent because the covariates are correlated.
For BH, the rank-specific thresholds \(kq/m\) are \(0.0125,\ 0.0250,\ 0.0375,\ 0.0500\). The largest rank, \(k=4\), passes because \(0.040\leq0.0500\). BH rejects all four hypotheses. This FDR guarantee is justified under independence or the specified positive-dependence condition, not by correlation alone.
For arbitrary dependence, the Benjamini–Yekutieli procedure (BY) uses the more conservative thresholds \(kq/(mH_m)\), where \(H_m=\sum_{j=1}^{m}1/j\). Here:
The four BY thresholds are approximately \(0.006,\ 0.012,\ 0.018,\ 0.024\). The first three pass: \(0.004\leq0.006\), \(0.012\leq0.012\), and \(0.018\leq0.018\). The fourth fails because \(0.040>0.024\). Thus the largest passing rank is 3, and BY rejects the first three hypotheses. BY provides FDR control under arbitrary dependence when the p-values are valid, but its penalty can reduce the number of discoveries.
The harmonic factor matters: it is \(H_m\), not a second factor of \(m\). In this example, omitting or miscalculating that factor would change the thresholds and could change the decisions.
Dependence, Sequential Searches, and Practical Diagnostics
Tests of pharmacokinetic covariates are often dependent. Covariates may be correlated, and tests may share subjects, model components, or a common likelihood. Dependence does not invalidate Bonferroni or Holm if the individual p-values are valid and the family is correctly specified. For BH, dependence must meet its conditions for the usual FDR guarantee; BY is a general-dependence alternative, usually at the cost of power.
A different problem arises when the family itself is created by the data. In forward addition, backward elimination, or bidirectional search, later tests may be chosen because of earlier p-values or model comparisons. Applying Holm or BH only to the tests that happened to be reported does not automatically restore their advertised error control. The standard guarantees concern a defined family of valid tests, not an arbitrary subset selected after inspecting the same data. A correction for dependence is not, by itself, a correction for adaptive selection.
For a screening plan, record the candidate hypotheses and the intended error criterion before interpreting results. If tests are performed conditionally along a search path, describe that path and avoid presenting ordinary fixed-family adjusted thresholds as guaranteed control unless the testing procedure justifies it. A prespecified family of tests from a common model can still have dependent p-values; this is different from changing the family in response to observed results.
Useful diagnostics include checking that the number \(m\) matches the actual family, that the hypotheses and p-values are valid for the fitted models, and that the ranking and stopping rule were implemented correctly. For BH or BY, verify every rank threshold and identify the largest passing rank. For Holm, verify that testing stops at the first failed comparison. Also report which covariates were tested, how tests were grouped into families, the chosen \(\alpha\) or \(q\), and whether the procedure was prespecified.
| Procedure | Target | Dependence condition for stated guarantee | Typical implication |
|---|---|---|---|
| Bonferroni | FWER | Arbitrary dependence; valid marginal p-values | Simple, often conservative cutoff |
| Holm | FWER | Arbitrary dependence; valid marginal p-values | Step-down and at least as powerful as Bonferroni |
| Benjamini–Hochberg | FDR | Independence or appropriate positive dependence | Can yield more screening discoveries |
| Benjamini–Yekutieli | FDR | Arbitrary dependence; valid marginal p-values | More conservative than BH |
These procedures answer different questions. FWER control is suited to settings where any false covariate claim is costly. FDR control can be useful for exploratory screening, where a controlled expected share of false discoveries is acceptable. An FDR target of 0.05 does not assert that each selected covariate has a 5% chance of being false, nor does it mean that 5% of the selected covariates in one particular analysis must be false.
Adjusted thresholds also do not establish clinical importance, causal effects, predictive improvement, or stability across alternative model specifications. A statistically significant covariate may have a small effect or be sensitive to modeling choices. Conversely, a non-rejection is not proof of no covariate effect. Multiplicity procedures regulate a specified type of statistical error; scientific interpretation still depends on effect estimates, uncertainty, diagnostics, and the study question.
Common Mistakes and Takeaways
- Using \(\alpha/m\) for every procedure. That is Bonferroni’s threshold, not Holm’s step-down sequence or BH’s rank-based rule.
- Continuing after a Holm failure. Holm stops at the first failed comparison; later thresholds are not applied to continue rejecting.
- Assuming BH works under any dependence. Its usual guarantee needs independence or an appropriate positive-dependence condition; BY is available for arbitrary dependence.
- Correcting only the reported tests after a search. A post hoc family may not have the fixed-family guarantees, especially when tests were selected using the same data.
- Equating FDR with an individual probability. FDR is an expected proportion among rejections over repeated applications, not a posterior probability for a particular selected effect.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Six valid p-values are tested as one prespecified family at \(\alpha=0.05\). What is the Bonferroni cutoff?
- For \(m=4\), \(\alpha=0.05\), and ordered p-values \(0.004,\ 0.014,\ 0.020,\ 0.30\), how many hypotheses does Holm reject?
- For \(m=5\) and \(q=0.10\), what is the BH threshold at rank 3?
- Why might BY be preferred to BH when covariate tests are dependent, and what is the cost?
- Why does applying an adjustment only to the covariates selected by a data-dependent search fail to guarantee the usual fixed-family error control?
Solutions. (1) \(0.05/6=0.00833\), so the Bonferroni cutoff is approximately 0.00833. (2) Holm compares \(0.004\leq0.0125\), then \(0.014\leq0.0167\), then \(0.020>0.025\) is false—more carefully, the third threshold is \(0.05/(4-3+1)=0.025\), so \(0.020\leq0.025\) passes. The fourth comparison is \(0.30>0.05\), so Holm rejects the first three hypotheses. (3) The rank-3 threshold is \(3(0.10)/5=0.060\). (4) BY controls FDR under arbitrary dependence when p-values are valid; its harmonic-factor adjustment makes it more conservative and can reduce rejections. (5) Selection based on the same data changes the tested family and can make the reported p-values’ ordinary fixed-family guarantees inapplicable; the adjustment alone does not account for that selection.