Why Repeated Random-Effects Tests Need a Plan
A random-effects structure is often built or reduced through a series of comparisons: test whether a random slope is needed, then assess another variance component, or compare a sequence of candidate structures. Each likelihood-ratio test may be individually calibrated to its null distribution, including a boundary reference distribution where appropriate. But if the analyst performs several tests and retains any effect with a nominally small p-value, the chance of selecting at least one effect that is truly absent can exceed the advertised level.
The issue is not unique to likelihood-ratio tests. It is the repeated opportunity to make a false rejection. In random-effects selection, it is especially important to separate two questions: whether each individual test has been calibrated correctly, and whether the procedure using several tests controls the overall error rate. A valid boundary-adjusted p-value for each comparison does not, by itself, correct for repeated testing.
For a random-effects selection problem, the family might consist of tests for several prespecified variance components or candidate random-effect terms. The null hypothesis for one such test could state that a particular variance component is zero. The alternative states that it is positive. The test statistic and its p-value must use an appropriate likelihood comparison and null calibration for that particular test. The earlier material on boundary likelihood-ratio tests addresses those individual calibrations.
Before applying a multiplicity procedure, ensure that comparisons are meaningful: use the same observed response data, specify which fixed effects are held constant, and use a likelihood method appropriate to the models being compared. For example, comparisons of random-effects structures with the same fixed-effects structure can use a suitable restricted likelihood comparison. Multiplicity adjustment cannot repair an invalid or mismatched individual test.
How Repetition Inflates Error
Suppose \(m\) independent null hypotheses are all true, and each is tested at level \(\alpha\). The probability of making no false rejection is \((1-\alpha)^m\), so the probability of at least one false rejection is
For \(m=3\) tests at \(\alpha=0.05\), this is \(1-(0.95)^3=0.1426\), not 0.05. Independence is used for this exact calculation, not for the general conclusion. When tests are dependent, the exact FWER changes, but repeatedly using level 0.05 does not generally guarantee that the overall FWER is 0.05.
Worked Example: Three Nominal Tests and a Bonferroni Threshold
A hypothetical analyst considers three prespecified random-effect additions. Each individual test has been calibrated, and the analyst plans to reject if its p-value is at most 0.05. If, for illustration, all three null hypotheses are true and the tests are independent, the probability of at least one false selection is
Thus the procedure has a 14.3% chance of at least one false rejection under this configuration. To control the FWER at 0.05 without requiring independence, the Bonferroni procedure tests each hypothesis at
Suppose the three calibrated p-values are 0.012, 0.031, and 0.044. Only 0.012 is at most 0.01667, so only the first null hypothesis is rejected. This procedure is valid under arbitrary dependence if each individual p-value is valid under its null. It can be conservative, especially when the tests are strongly dependent.
Bonferroni and Holm Adjustments
For \(m\) tests with valid p-values \(p_1,\ldots,p_m\), the Bonferroni rule rejects test \(j\) when \(p_j\leq \alpha/m\). Its justification is the union bound. If \(H_j\) is true, the probability of falsely rejecting it is at most \(\alpha/m\); therefore, the probability of falsely rejecting any true null is at most the sum of those probabilities, which is no more than \(m(\alpha/m)=\alpha\). The bound remains valid if some hypotheses are false, since the number of true nulls is at most \(m\).
Holm’s procedure is never less powerful than applying the same Bonferroni cutoff to every test: its first threshold is \(\alpha/m\), but the thresholds increase as the procedure proceeds. Its FWER justification can be seen by considering the first true null in the sorted list. If any true null is rejected, that first true null must pass its step-down threshold. If it is at rank \(r\), all \(m_0\) true nulls occur at rank \(r\) or later, so \(m-r+1\geq m_0\). Its threshold is therefore at most \(\alpha/m_0\). The probability that any of the \(m_0\) true-null p-values is at most \(\alpha/m_0\) is at most \(m_0(\alpha/m_0)=\alpha\), by the union bound. This establishes strong FWER control.
Worked Example: Applying Holm to Three Random-Effect Tests
In a hypothetical analysis, three prespecified and appropriately calibrated tests assess whether three candidate random-effect terms can be omitted. Their p-values are 0.009, 0.024, and 0.041. With \(\alpha=0.05\), order them from smallest to largest and apply Holm’s thresholds:
All three comparisons pass, so all three null hypotheses are rejected by this family-wise procedure. This is not the same as claiming that the probability each null is true is small. Rather, under the procedure’s assumptions, the probability of rejecting one or more true null hypotheses in the specified family is controlled at 0.05.
For reference, the Holm-adjusted p-values are 0.027, 0.048, and 0.048. They are obtained from the ordered p-values using the step-down multipliers \(3,2,1\) and taking cumulative maxima: \(3(0.009)=0.027\); \(\max(0.027,2(0.024))=0.048\); and \(\max(0.048,1(0.041))=0.048\). Each adjusted value is at most 0.05, matching the step-down decisions.
Prespecified Sequential Strategies
Not every analysis needs to test all candidate terms symmetrically. A sequential strategy can be useful when the order of tests has a defensible scientific or design-based rationale. Two approaches are alpha spending and fixed-sequence testing. In either case, the testing order or allocation should be stated before examining the results; choosing the order after seeing which p-values are smallest changes the procedure.
With alpha spending, assign test \(j\) a threshold \(\alpha_j\) in advance, with \(\sum_{j=1}^m\alpha_j\leq\alpha\). Reject a tested null only if its p-value is at most \(\alpha_j\). The union-bound argument gives FWER at most \(\alpha\), provided the tests and null hypotheses are defined in advance and the individual p-values are valid. If the procedure selects tests adaptively, their p-values must remain valid under that selection; merely assigning thresholds after the fact does not guarantee control.
Worked Example: A Prespecified Alpha-Spending Sequence
A hypothetical analysis defines four random-effect tests in advance and assigns them thresholds 0.020, 0.015, 0.010, and 0.005. The total is \(0.020+0.015+0.010+0.005=0.050\). The analysis proceeds in that order and stops at the first failure. Suppose the first two p-values are 0.012 and 0.018.
The first comparison passes because \(0.012\leq0.020\), so the first null hypothesis is rejected. The second comparison fails because \(0.018>0.015\); the sequence stops, and no later term is selected through this procedure. The decisions do not establish that the untested later effects are absent. Under the prespecified family and valid p-values, the chance of at least one false rejection is no more than the sum of the allocated thresholds, 0.05.
A fixed-sequence procedure is another option: prespecify an order, test each hypothesis at level \(\alpha\), and continue only if the current null is rejected. This can be more powerful than dividing \(\alpha\) across tests, but it makes later rejections conditional on rejecting every earlier hypothesis. Its FWER control follows because, if any true null is rejected, the first true null in the fixed order must have been rejected at level \(\alpha\). The probability of that event is at most \(\alpha\). This guarantee depends on fixing the order in advance; sorting hypotheses by observed p-values and then describing the result as a fixed sequence is not the same procedure.
A single prespecified joint test of a block of random effects is a different alternative. It addresses the joint null that the specified block is absent, rather than identifying which individual component is needed. If that joint null is rejected, further component-level testing is a separate inferential step and needs a prespecified multiplicity strategy. The joint test should not be treated as though it automatically licenses an unlimited series of nominal-level follow-up tests.
Selection, Dependence, and Diagnostics
Random-effect tests are often dependent because they use the same observations and compare related models. Bonferroni and Holm remain valid under arbitrary dependence when their input p-values are valid. Dependence can make these procedures conservative, but it does not justify ignoring multiplicity. More specialized procedures may exploit a known dependence structure, but their calibration must match the actual testing plan.
A practical diagnostic is to write down the complete set of candidate hypotheses and the rule for moving among them before fitting the comparisons. Then check that the reported tests match that plan. Look for these warning signs:
- Only significant comparisons are reported, with no account of how many candidate terms were considered.
- The testing order was chosen after reviewing the p-values, but the analysis is described as prespecified sequential testing.
- Tests use different observations, fixed-effects structures, or likelihood criteria without explaining how the comparisons remain valid.
- A p-value calibrated for a single boundary test is treated as though it already controls error across a series of tests.
- A component not reached after a stopping rule is reported as proven unnecessary.
Multiplicity control also has limits. FWER control bounds the chance of one or more false rejections; it does not guarantee that the selected random-effects structure is the unique true structure, that variance estimates are precise, or that a selected model will predict well. Adjustment can reduce false selections at the cost of power. Decisions should therefore also reflect the scientific purpose, design, and adequacy of the fitted model.
Choosing and Reporting a Strategy
| Strategy | When it fits | Main caution |
|---|---|---|
| Bonferroni | A simple, transparent adjustment for a prespecified family | Can be conservative when tests are dependent |
| Holm | A general step-down procedure for a fixed family of valid p-values | Requires a clearly defined family and valid p-values |
| Alpha spending | A prespecified sequence with explicit thresholds for its tests | Thresholds and hypotheses should not be chosen after inspecting results |
| Fixed sequence | A scientifically justified order where later testing is meaningful only after earlier rejection | Later hypotheses cannot be rejected after an earlier failure |
| Joint block test | The primary question concerns a specified group of random effects together | Does not identify which component matters without further planned inference |
A clear report states the candidate family, the individual test calibration, the multiplicity strategy, and the stopping or rejection rule. For example: “Three prespecified variance-component nulls were tested using boundary-calibrated likelihood-ratio p-values; Holm adjustment at family-wise level 0.05 rejected the three nulls.” This describes the procedure without claiming that the selected structure is certain or that any effect is causal.
Check Your Understanding
Work through each question, then compare your reasoning with the solutions below.
- Four independent tests are each performed at level 0.05 when all four null hypotheses are true. Calculate the chance of at least one false rejection, and give the Bonferroni per-test threshold for FWER 0.05.
- For four prespecified tests, the ordered p-values are 0.010, 0.030, 0.040, and 0.20. Apply Holm’s procedure at 0.05.
- A fixed-sequence plan has p-values 0.03, 0.08, and 0.001, tested in that prespecified order at 0.05. Which null hypotheses are rejected?
- Why does a correctly calibrated p-value for each boundary likelihood-ratio test not, by itself, control the FWER across a sequence of random-effects tests?
Solutions. (1) The FWER under independence is \(1-(1-0.05)^4=1-0.81450625=0.1855\), or 18.55%. The Bonferroni threshold is \(0.05/4=0.0125\). (2) The first p-value passes \(0.05/4=0.0125\). The second fails \(0.05/3=0.01667\), so Holm stops; only the first null is rejected. (3) The first null is rejected because \(0.03\leq0.05\). The second fails because \(0.08>0.05\), so testing stops and the third null is not rejected despite its small p-value. (4) Individual calibration bounds the false-rejection probability for one test under its null. It does not bound the probability of at least one false rejection across multiple opportunities; an adjustment or a prespecified sequential rule is needed for that overall guarantee.