Why a Family of Tests Needs Separate Attention
A GLMM analysis may test several fixed-effect coefficients, contrasts, or predictor terms. Each test can be reported with a p-value, but interpreting many unadjusted p-values at once changes the chance of making at least one false rejection. This is a multiplicity problem: the relevant error rate depends not only on each test, but also on the family of hypotheses considered together.
The family should be defined by the scientific question and analysis plan. For example, a set of prespecified treatment contrasts for one outcome could form a family. A separate set of tests for unrelated outcomes might form another family, if that separation was justified in advance. Defining many small families after looking at results can conceal the amount of testing and weaken error-rate control.
FWER control limits the probability of making even one false rejection in the family. FDR control instead limits the expected proportion of false rejections among the rejected hypotheses, with the proportion defined as zero when there are no rejections. FDR control does not mean that every rejected hypothesis has a specified probability of being false.
Conditions for Adjusting GLMM Tests
Multiplicity procedures adjust the decisions or p-values from individual tests; they do not make an invalid individual test valid. A generic GLMM has clustered responses \(Y_{ij}\), where \(i\) indexes clusters and \(j\) indexes observations within a cluster. One common form is
Here \(\beta\) contains the fixed effects being tested, \(b_i\) is a cluster-specific random effect, and \(g\) is the link function. The model and test must be appropriate for the outcome and design: for example, the conditional response distribution and link should be specified correctly, the random-effects structure should be defensible, and observations should have the modeled conditional independence structure. The chosen fixed-effect tests must also have p-values calibrated under their null hypotheses. Wald p-values generally rely on an adequate large-sample approximation; likelihood-ratio p-values rely on regularity conditions for their reference distribution.
Write \(p_j\) for the p-value for \(H_{0j}\). A p-value is valid under its null when \(P(p_j\le t)\le t\) for every fixed \(t\) between zero and one. Exact uniformity is not required; conservative p-values satisfy this inequality. Procedures such as Bonferroni and Holm need this marginal validity, but do not require independence among the tests. Dependence is common when tests use estimates from the same fitted GLMM.
Bonferroni and Holm Control the Familywise Error Rate
The Bonferroni procedure rejects \(H_{0j}\) when \(p_j\le\alpha/m\), where \(\alpha\) is the desired FWER bound. Equivalently, it reports adjusted p-values \(\min(1,mp_j)\). Its validity follows directly from the union bound:
where \(m_0\le m\) is the number of true null hypotheses. No independence assumption appears in this argument. The cost is that Bonferroni can be conservative, particularly when tests are numerous or highly correlated.
Holm’s step-down procedure is usually at least as powerful while retaining strong FWER control under arbitrary dependence. Order the p-values as \(p_{(1)}\le\cdots\le p_{(m)}\). Compare \(p_{(i)}\), in order, with \(\alpha/(m-i+1)\). Reject sequentially while the p-value is no larger than its threshold; stop at the first failure and do not reject that or any later hypothesis. The equivalent adjusted p-value for rank \(i\) is
The running maximum makes the adjusted p-values nondecreasing in rank, as required for coherent step-down decisions.
To justify the result, suppose at least one true null is rejected, and let \(i\) be the rank of the first true null among the rejected hypotheses. All hypotheses at earlier ranks are false nulls, so all \(m_0\) true nulls occur at rank \(i\) or later. Thus \(m-i+1\ge m_0\). The true null at rank \(i\) was rejected, so
Consequently, any false rejection implies that at least one true-null p-value is at most the fixed threshold \(\alpha/m_0\). Applying validity at that fixed threshold and then the union bound gives
This argument does not apply validity at a threshold chosen from the observed rank. It first bounds every possible true-null rejection by the same nonrandom threshold, which is essential to the justification.
Worked Example: Holm Adjustment for Five Fixed-Effect Tests
Suppose a hypothetical GLMM analysis tests five prespecified fixed-effect contrasts. Their p-values, sorted from smallest to largest, are \(0.004,\ 0.018,\ 0.031,\ 0.049,\ 0.200\). Use Holm’s procedure with \(\alpha=0.05\).
The sequential thresholds are \(0.05/5=0.0100,\ 0.05/4=0.0125,\ 0.05/3=0.0167,\ 0.05/2=0.0250,\) and \(0.0500\). The first p-value passes because \(0.004\le0.0100\). The second fails because \(0.018>0.0125\), so testing stops. Only the hypothesis associated with the smallest p-value is rejected.
The adjusted p-values are found from the running maxima: \(5(0.004)=0.020\); \(\max(0.020,4(0.018))=0.072\); then \(\max(0.072,3(0.031))=0.093\); then \(\max(0.093,2(0.049))=0.098\); and finally \(\max(0.098,0.200)=0.200\). Thus the adjusted p-values are \(0.020,\ 0.072,\ 0.093,\ 0.098,\ 0.200\). At the 0.05 FWER level, only the first contrast is significant after adjustment.
False-Discovery-Rate Adjustment
When the analysis involves many related tests and the goal is to identify a useful set of signals, controlling the FDR may be more appropriate than requiring a small chance of any false positive. The Benjamini–Hochberg (BH) procedure orders the p-values and finds the largest rank \(k\) satisfying
It rejects the hypotheses at ranks \(1,\ldots,k\). If no rank satisfies the inequality, it rejects none. For independent valid p-values, BH controls the FDR at or below \(\alpha\); it also does so under certain positive dependence conditions, commonly expressed as positive regression dependence on a subset (PRDS) for the true-null p-values. Shared GLMM estimates can induce dependence, so independence should not be assumed merely because the tests have different labels.
BH adjusted p-values, often called q-values in this procedure, are computed as
The minimum taken from rank \(i\) onward ensures that adjusted values do not decrease as the ordered raw p-values increase. Under arbitrary dependence, a more conservative option is the Benjamini–Yekutieli (BY) procedure: replace the BH threshold \(k\alpha/m\) with \(k\alpha/(mH_m)\), where \(H_m=\sum_{r=1}^{m}1/r\). This controls FDR under arbitrary dependence for valid p-values, but can have substantially less power.
Worked Example: Applying Benjamini–Hochberg
Consider six hypothetical tests of fixed-effect terms in a GLMM, with ordered p-values \(0.002,\ 0.009,\ 0.014,\ 0.028,\ 0.041,\ 0.200\). At \(\alpha=0.05\), the BH thresholds are \(0.00833,\ 0.0167,\ 0.0250,\ 0.0333,\ 0.0417,\ 0.0500\), respectively.
The first five p-values meet their corresponding thresholds, including \(0.041\le0.0417\) at rank five. The sixth does not. The largest passing rank is therefore \(k=5\), and BH rejects the first five hypotheses. This conclusion assumes the p-values are valid and their dependence meets the conditions for BH control.
For the adjusted values, calculate \(mp_{(i)}/i\): \(0.0120,\ 0.0270,\ 0.0280,\ 0.0420,\ 0.0492,\ 0.200\). Taking the minimum from each rank onward gives the same sequence, capped at one. The adjusted p-values are therefore \(0.0120,\ 0.0270,\ 0.0280,\ 0.0420,\ 0.0492,\ 0.200\). At the 0.05 FDR level, the first five are rejected; this does not imply that each rejected test has a 5% probability of being false.
Choosing a Procedure and Diagnosing Trouble
The choice of adjustment should follow the inferential goal, not which method produces more rejections. Holm is suitable when the aim is to limit the chance of any false rejection in a family, with no need to assume independence. BH is often used when a larger set of discoveries is the goal and the relevant dependence conditions are reasonable. BY offers FDR control under arbitrary dependence, at the cost of a more stringent threshold. Bonferroni is simple and valid under arbitrary dependence, but Holm generally dominates it in power while preserving FWER control.
| Procedure | Target error rate | Dependence condition | Typical use |
|---|---|---|---|
| Bonferroni | FWER | Arbitrary dependence | Simple familywise protection |
| Holm | FWER | Arbitrary dependence | Familywise protection with step-down testing |
| Benjamini–Hochberg | FDR | Independence or suitable positive dependence | Screening a set of related hypotheses |
| Benjamini–Yekutieli | FDR | Arbitrary dependence | FDR protection when dependence assumptions are unavailable |
A useful check is to inspect the fitted model, test definitions, and p-value calibration before applying an adjustment. Sparse binary outcomes, few clusters, or unstable estimates can make standard large-sample p-values unreliable; multiplicity correction cannot repair that problem. Likewise, adjusting p-values does not fix a misspecified random-effects structure, account automatically for unplanned data-dependent selection, or establish that an estimated effect is practically important. Report the family, the adjustment method, the target level, and the adjusted results.
Multiplicity adjustment is therefore a decision layer built on top of valid tests. First establish a defensible GLMM and valid fixed-effect p-values; then state the family and the error rate relevant to the scientific aim; finally apply and report the selected procedure.
Worked Example: Different Goals, Different Conclusions
Suppose four prespecified fixed-effect contrasts from a hypothetical clustered binary-outcome GLMM have ordered p-values \(0.006,\ 0.021,\ 0.038,\ 0.110\). Assume the individual tests are valid, and suppose BH’s dependence condition is considered reasonable. Compare Holm and BH at \(\alpha=0.05\).
For Holm, the thresholds are \(0.0125,\ 0.0167,\ 0.0250,\ 0.0500\). The first value passes, but \(0.021>0.0167\), so only the first contrast is rejected. The Holm adjusted p-values are \(0.024,\ 0.063,\ 0.076,\ 0.110\), obtained by taking running maxima of \(4(0.006),3(0.021),2(0.038),1(0.110)\).
For BH, the thresholds are \(0.0125,\ 0.0250,\ 0.0375,\ 0.0500\). The second p-value passes and the third does not, so the largest passing rank is two. BH rejects the first two contrasts. The BH adjusted p-values are \(0.024,\ 0.042,\ 0.0507,\ 0.110\), using the running minima from the last rank backward. Both conclusions are consistent with their goals: Holm is stricter about the chance of any false rejection, while BH controls the expected false-discovery proportion under its assumptions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For \(m=8\) tests and \(\alpha=0.04\), what is the Bonferroni per-test threshold?
- In Holm’s procedure, if a sorted p-value fails its rank-specific threshold, what happens to hypotheses at later ranks?
- What does FDR control at level 0.05 mean, and what does it not mean?
- Why can BH not be assumed valid under arbitrary dependence, and what procedure provides FDR control under arbitrary dependence?
Solutions. (1) The threshold is \(\alpha/m=0.04/8=0.00500\). (2) The step-down procedure stops, so that hypothesis and all later-ranked hypotheses are not rejected. (3) Under the procedure’s assumptions, the expected proportion of false rejections among all rejections is at most 0.05; it does not say that every rejected hypothesis has a 5% chance of being false or guarantee a maximum proportion in each realized analysis. (4) BH’s FDR guarantee requires independence or suitable positive dependence; under arbitrary dependence, the more conservative Benjamini–Yekutieli procedure provides FDR control for valid p-values.