Common Misreadings of Adjusted P-Values and Q-Values

Why the Name “Adjusted” Is Not Enough

A genome-wide association analysis can produce a very large collection of p-values, one for each tested variant or hypothesis. Adjustments help address the errors that can arise when many hypotheses are tested, but different adjustments target different quantities. A small adjusted p-value is not automatically the same kind of evidence under every method, and a q-value is not a probability that an individual association is false.

The first task is to identify the error rate the procedure targets. Bonferroni and Holm procedures target the family-wise error rate, the probability of making at least one false rejection in a family of tests. The Benjamini–Hochberg (BH) procedure targets the false discovery rate, the expected proportion of false rejections among all rejections. Those are different guarantees, so adjusted values from the two approaches should not be interpreted interchangeably.

Definition: Let \(R\) be the number of hypotheses rejected and \(V\) the number of those rejections that are false. The family-wise error rate is \(P(V\geq 1)\). The false discovery rate is \(E\left[V/\max(R,1)\right]\), where the ratio is defined as zero when \(R=0\).

A procedure controlling the family-wise error rate at level \(\alpha\) limits the probability of one or more false rejections to at most \(\alpha\), under its assumptions. A procedure controlling the false discovery rate at target level \(q\) limits the expected false-discovery proportion to at most \(q\), under its assumptions. Neither guarantee says that the realized error proportion in every particular analysis will be below the target.

Bonferroni and Holm Adjusted P-Values

Suppose \(m\) hypotheses are tested, with unadjusted p-values \(p_1,\ldots,p_m\). A Bonferroni-adjusted p-value for test \(i\) is the smaller of 1 and \(m p_i\). Rejecting when that adjusted value is at most \(\alpha\) is equivalent to rejecting when the original p-value is at most \(\alpha/m\).

p_{i,\mathrm{Bonf}}=\min(1,mp_i)

The family-wise error guarantee does not require independent tests. If \(H_i\) is true, a valid p-value satisfies \(P(p_i\leq t)\leq t\) for each threshold \(t\). The probability of at least one false rejection is at most the sum of the probabilities of false rejections across the true null hypotheses. This is the union bound:

P(V\geq 1) \leq \sum_{i\in I_0}P(p_i\leq \alpha/m) \leq |I_0|\frac{\alpha}{m} \leq \alpha,

where \(I_0\) is the set of true null hypotheses and \(|I_0|\leq m\). This explains why Bonferroni controls the family-wise error rate even when test statistics are dependent. Its simplicity can come at a cost: the same threshold is applied to every test, which can be conservative.

Holm’s step-down procedure also controls the family-wise error rate under arbitrary dependence of valid p-values, and it is never less powerful than the single-step Bonferroni procedure. To define its adjusted values, sort the p-values from smallest to largest, denoting them \(p_{(1)}\leq\cdots\leq p_{(m)}\). The Holm adjusted p-value at rank \(i\) is the cumulative maximum, through that rank, of the Bonferroni-scaled ordered p-values, capped at 1:

p_{(i),\mathrm{Holm}} = \min\left(1,\max_{1\leq k\leq i}\{(m-k+1)p_{(k)}\}\right).

The cumulative maximum ensures that adjusted values do not decrease as the ordered raw p-values increase. At a chosen level \(\alpha\), the step-down procedure compares \(p_{(i)}\) with \(\alpha/(m-i+1)\), stopping at the first failure. Equivalently, reject hypotheses whose Holm adjusted p-values are at most \(\alpha\).

Worked Example: Reading Bonferroni and Holm Values

Suppose a hypothetical analysis tests four variant–trait hypotheses and obtains ordered p-values \(0.003,\ 0.012,\ 0.040,\ 0.200\). Calculate both adjusted values and determine the rejections at family-wise level \(\alpha=0.05\).

Bonferroni. Multiply each raw p-value by \(m=4\), capping at 1: \(4(0.003)=0.012\), \(4(0.012)=0.048\), \(4(0.040)=0.160\), and \(4(0.200)=0.800\). Thus the adjusted values in order are \(0.012,\ 0.048,\ 0.160,\ 0.800\). At \(\alpha=0.05\), the first two hypotheses are rejected.

Holm. The rank-specific multipliers are 4, 3, 2, and 1. The scaled values are \(4(0.003)=0.012\), \(3(0.012)=0.036\), \(2(0.040)=0.080\), and \(1(0.200)=0.200\). These are already nondecreasing, so the cumulative maxima are \(0.012,\ 0.036,\ 0.080,\ 0.200\). At \(0.05\), Holm also rejects the first two hypotheses.

The second Holm adjusted value is \(0.036\), not \(0.048\), because Holm uses a smaller multiplier at the second rank. In this example, both procedures select the same two hypotheses, but their adjusted values differ. Each conclusion concerns family-wise error control, not the probability that an individual rejected association is false.

BH Adjusted Values and the Meaning of a Q-Value

The BH procedure orders the p-values and compares them with rank-specific thresholds. At a target level \(q\), it finds the largest rank \(k\) satisfying \(p_{(k)}\leq kq/m\), then rejects the hypotheses at ranks 1 through \(k\). Under independence, and under certain positive dependence conditions, the procedure controls the false discovery rate at or below \(q\).

A BH adjusted p-value, often called a BH q-value, gives the smallest target level at which that hypothesis would be rejected by the BH procedure. For ordered raw p-values, it is calculated by multiplying by \(m/i\) at each rank \(i\), then taking the minimum over that rank and all larger ranks, with a cap of 1:

q_{(i),\mathrm{BH}} = \min\left(1,\min_{j\geq i}\left\{\frac{m p_{(j)}}{j}\right\}\right).

The minimum over later ranks makes the adjusted values nondecreasing with rank. Comparing BH adjusted values with \(q\) gives the same rejection set as applying the rank-threshold procedure at \(q\). The name “q-value” is used in more than one way: BH adjusted p-values are often labelled q-values in software, while Storey’s q-values use an estimate of the proportion of true null hypotheses. A report should identify the method rather than relying on the label alone.

Worked Example: Calculating BH Q-Values

Suppose a hypothetical study tests \(m=5\) hypotheses and has ordered p-values \(0.001,\ 0.008,\ 0.020,\ 0.040,\ 0.300\). Calculate the BH adjusted values and identify the rejections at target \(q=0.05\).

First calculate \(mp_{(i)}/i\) at each rank: at ranks 1 through 5 these are \(5(0.001)/1=0.005\), \(5(0.008)/2=0.020\), \(5(0.020)/3\approx0.03333\), \(5(0.040)/4=0.050\), and \(5(0.300)/5=0.300\). Each value is already no larger than the value at the next rank, so taking the required minimum over its own and later ranks leaves the sequence unchanged. The BH adjusted values are therefore \(0.005,\ 0.020,\ 0.03333,\ 0.050,\ 0.300\).

At target \(q=0.05\), the first four adjusted values are at most \(0.05\), so BH rejects the first four hypotheses. This is an FDR-targeting result under the procedure’s dependence conditions. It does not promise that exactly 5% of these four discoveries are false, or that each discovery has a 5% chance of being false.

What the Values Do—and Do Not—Say

An adjusted p-value is best understood as a threshold quantity: it indicates the smallest error-rate level at which a specified multiple-testing procedure would reject that hypothesis. It is not a posterior probability, and it does not directly quantify the probability that the null hypothesis is true. A raw p-value also is not the probability that the null is true; adjustment changes the rejection rule, not that basic interpretation.

For a Bonferroni or Holm adjusted value of 0.02, the precise reading is that the hypothesis would be rejected at family-wise level 0.02 or any larger level, subject to the validity of the p-values and the stated testing family. It is not correct to say there is a 2% probability that this particular association is a false positive. The family-wise guarantee concerns the chance of at least one false rejection across the family in repeated applications.

For a BH adjusted value of 0.02, the reading is that this hypothesis enters the BH rejection set at target \(q=0.02\) or above, assuming the specified BH procedure and its applicable dependence conditions. FDR control is an expectation over repeated applications; it does not allocate a personal false-discovery probability to each selected variant. The realized false-discovery proportion in one analysis can be higher or lower than the target.

Worked Example: FDR Is Not a Per-Finding Probability

Imagine 100 hypothetical repetitions of a study, with exactly 10 discoveries selected in each repetition. Suppose exactly one of the 10 is false in each repetition. In every repetition the realized false-discovery proportion is \(1/10=0.10\), and the average false-discovery proportion across repetitions is also \(0.10\). This illustrates an FDR of 0.10 in this deliberately simple scenario.

It would still be incorrect to infer that each selected association has a 10% probability of being false. FDR summarizes the expected proportion of false results in a selected set across repetitions; it does not give an individual probability for any one result. In an actual study, the realized number of false discoveries is unknown, and an FDR guarantee does not identify which findings are false.

Assumptions, Scope, and Common Misreadings

The guarantees depend on valid input p-values and the conditions of the procedure. Bonferroni and Holm family-wise error control use the validity of each null p-value and do not require independence among tests. The standard BH FDR guarantee applies under independent tests and also under certain positive dependence structures; arbitrary dependence is not covered by that standard guarantee. A dependence-robust correction is more conservative. The fact that an adjusted value can be computed does not by itself establish that the assumptions for its error-rate guarantee hold.

Nor does multiplicity adjustment correct other problems in an analysis. It does not repair confounding, population structure, poor phenotype measurement, or invalid null calibration. An adjusted value is conditional on the p-values supplied to the procedure and the family of tests being adjusted. Defining that family and checking calibration remain important parts of interpreting genetic association evidence.

Misreading: “A BH q-value of 0.05 means this association has a 5% probability of being false.” Correction: It is a BH rejection threshold associated with a procedure-level FDR target, not an individual probability.
Misreading: “A Bonferroni-adjusted value of 0.03 means a 3% chance of at least one false positive in this completed analysis.” Correction: It is the minimum family-wise level at which that hypothesis would be rejected; the FWER guarantee concerns repeated use of the procedure.
Misreading: “Every adjustment controls the same error rate.” Correction: Bonferroni and Holm control family-wise error; BH targets false discovery rate under its assumptions.
Misreading: “Because software reports q-values, the method must be BH.” Correction: State whether values are BH adjusted p-values or a different q-value estimate, such as Storey’s.

Key takeaway: Always name the procedure, the error rate it targets, and the conditions behind its guarantee. Treat adjusted p-values and q-values as procedure-specific rejection thresholds—not as probabilities that individual genetic findings are false.

Check Your Understanding

Answer each question by naming the relevant error rate and stating what the adjusted value does and does not mean.

What is the difference between family-wise error rate and false discovery rate?
For four tests, a raw p-value is 0.01. What is its Bonferroni-adjusted p-value, and what does it mean at a family-wise level of 0.05?
Why can Holm adjusted values be smaller than Bonferroni adjusted values while both procedures control family-wise error?
What does a BH adjusted p-value of 0.04 indicate about the BH rejection rule, and why is it not the probability that the finding is false?
Which dependence conditions support the standard BH FDR guarantee, and why does computing a BH value alone not establish that guarantee?