Why the Dependence Condition Matters
In a multi-arm trial, treatment-versus-control test statistics often share the control group. Their p-values can therefore be dependent even when each test is individually valid. Applying a multiple-testing procedure that requires a dependence condition unsupported by the design or analysis can undermine the familywise error rate (FWER), the probability of rejecting at least one true null hypothesis in the family.
Hochberg’s procedure is an appealing step-up method: it can be more powerful than procedures designed to work under arbitrary dependence. But its validity is not guaranteed just because each null p-value is marginally valid. The joint behavior of the p-values matters. This tutorial makes that point concrete with a global-null construction in which each p-value is exactly uniform, but Hochberg’s FWER is greater than the nominal level.
Hochberg’s Step-Up Rule
Let \(p_1,\ldots,p_m\) be the p-values for the hypotheses in a fixed family, and write their ordered values as \(p_{(1)}\leq \cdots \leq p_{(m)}\). At familywise level \(\alpha\), Hochberg compares \(p_{(i)}\) with \(\alpha/(m-i+1)\), starting from the largest ordered p-value. It finds the largest \(i\) satisfying the comparison and rejects the hypotheses corresponding to \(p_{(1)},\ldots,p_{(i)}\). If no comparison is satisfied, it rejects none.
The procedure’s step-up feature can make it less conservative than Bonferroni or Holm, but the thresholds alone do not establish error control. A standard sufficient setting for Hochberg’s strong FWER control is independence of the p-values under the null configuration. There are also dependence settings under which Simes-type inequalities support validity. When those conditions are not justified, one should not assume that Hochberg controls FWER.
Worked Example: Applying Hochberg Without Making a Validity Claim
Suppose three hypotheses have p-values \(0.008,\ 0.021,\ 0.047\), and the familywise level is \(\alpha=0.05\). Order the values: \(p_{(1)}=0.008,\ p_{(2)}=0.021,\ p_{(3)}=0.047\). Hochberg’s thresholds are \(0.05/3=0.01667,\ 0.05/2=0.025,\) and \(0.05\).
Since \(p_{(3)}=0.047\leq 0.05\), the largest qualifying index is \(i=3\), so the procedure rejects all three hypotheses. This is the mechanical decision for these p-values. Whether using Hochberg guarantees FWER control is a separate question about the joint distribution of p-values under the nulls.
A Dependence Construction With Uniform Margins
The following hypothetical construction uses three true null hypotheses and \(\alpha=0.05\). It is designed to separate marginal validity from joint validity: each coordinate has an exactly uniform distribution on \((0,1)\), but the coordinates are dependent in a way that makes Hochberg reject too often. The exact fractions matter for verifying the uniform margins.
Generate the vector of three p-values using one of three cases:
- Case A, with probability \(1/20=0.05\): choose one of the three coordinates uniformly at random. Give the chosen coordinate a value drawn uniformly from \((0,1/60]\). Give the other two values drawn uniformly from \((1/40,1)\).
- Case B, with probability \(1/80=0.0125\): choose one of the three pairs uniformly at random. Give both coordinates in the pair values drawn uniformly from \((1/60,1/40]\). Give the remaining coordinate a value drawn uniformly from \((1/40,1)\).
- Case C, with probability \(15/16=0.9375\): give all three coordinates values drawn independently and uniformly from \((1/40,1)\).
In Cases A and B, the chosen coordinate or pair is selected symmetrically. For any fixed coordinate, the probability of receiving a value in \((0,1/60]\) is \((1/20)(1/3)=1/60\), exactly the interval’s width. Its probability of receiving a value in \((1/60,1/40]\) is \((1/80)(2/3)=1/120\), also exactly that interval’s width.
The remaining probability that this coordinate lies in \((1/40,1)\) is \((1/20)(2/3)+(1/80)(1/3)+15/16=1/30+1/240+15/16=39/40\). This equals the width \(1-1/40=39/40\). Within each of the three intervals, the coordinate is uniform conditional on being in that interval. Its marginal density is therefore one throughout \((0,1)\): each interval’s probability mass equals its width. Thus every p-value is exactly uniform under its null, even though the three p-values are not jointly independent.
Worked Example: Calculating the FWER in the Construction
Under the complete null, any Hochberg rejection is a familywise error. In Case A, at least one p-value is at most \(1/60=0.01667\), which is \(\alpha/3\). If no larger-index comparison already triggers rejection, the smallest p-value therefore meets Hochberg’s first threshold. So Case A always produces at least one rejection.
In Case B, two p-values are at most \(1/40=0.025=\alpha/2\). Thus \(p_{(2)}\leq\alpha/2\), and Hochberg rejects at least two hypotheses. Case B also always produces a familywise error.
In Case C, all three p-values exceed \(1/40=0.025\). Neither the \(p_{(1)}\leq\alpha/3\) nor the \(p_{(2)}\leq\alpha/2\) comparison can succeed. Hochberg rejects only if \(p_{(3)}\leq 0.05=1/20\). Each Case C coordinate is independently uniform on \((1/40,1)\), an interval of width \(39/40\). The probability for one coordinate to fall in \((1/40,1/20]\) is \((1/20-1/40)/(39/40)=1/39\). Consequently, the probability that all three are at most \(1/20\), conditional on Case C, is \((1/39)^3\).
The cases are disjoint, so the unconditional FWER is
This is greater than \(0.05\). The construction demonstrates that uniform marginal null p-values alone do not ensure Hochberg’s FWER control. The excess is not a rounding artifact: the cutoffs, case probabilities, interval widths, and resulting marginal probabilities match exactly.
Why This Is a Simes-Type Failure
Under the complete null, Hochberg rejects at least one hypothesis exactly when at least one of its ordered comparisons succeeds. For three tests, this event is
This is the rejection event used by the Simes global test. Under independence, its probability under the complete null is controlled at level \(\alpha\). Dependence can change that probability, even when each \(p_j\) remains uniform. In the construction above, the deliberately clustered low values make the event occur with probability greater than \(\alpha\). This supplies a direct diagnostic of the problem: if the joint null distribution does not satisfy the needed bound on this event, complete-null control fails. A failure under the complete null is already enough to rule out a claim of strong FWER control.
Strong control means controlling FWER for every configuration of true and false null hypotheses, not only when all nulls are true. Establishing strong control under a dependence assumption therefore requires that the relevant condition hold across the null configurations covered by the procedure’s guarantee. Do not infer strong control merely from checking one observed set of p-values or from noting that every individual test has a valid p-value.
What to Check in a Multi-Arm Trial
In a trial with several treatment-versus-control comparisons, the shared control observations can induce dependence among test statistics and p-values. Additional features—such as endpoint relationships or repeated measurements—can also affect their joint distribution. The dependence is not necessarily harmful in every design, but it must be addressed rather than presumed harmless.
When considering Hochberg, ask whether the statistical model and the planned comparisons support the procedure’s stated dependence condition. The relevant question concerns the joint distribution under the null hypotheses, not whether the observed p-values happen to look correlated. Observed correlations are noisy, and an estimated correlation near zero does not prove independence of null p-values. Likewise, knowing that comparisons share participants warns that dependence is plausible, but does not by itself quantify how much FWER will change.
Useful checks include confirming how test statistics share observations, reviewing the joint distribution implied by the analysis model, and using design-based or simulation calculations under null configurations when the dependence structure can be specified. If that work cannot substantiate the required condition, choose a method with a guarantee that matches what is known.
Worked Example: Choosing a Procedure When Dependence Is Unsupported
Suppose a hypothetical trial has three treatment-control p-values \(0.018,\ 0.024,\ 0.049\), with \(\alpha=0.05\). Hochberg’s largest comparison succeeds because \(0.049\leq0.05\), so it rejects all three. If its dependence conditions are unsupported, that decision cannot be presented as having guaranteed 5% FWER.
Holm’s step-down procedure is a dependence-robust alternative when the individual null p-values are valid. It first compares the smallest p-value with \(\alpha/3=0.01667\). Here \(0.018>0.01667\), so Holm stops and rejects none. Bonferroni’s single-step rule also compares each p-value with \(0.01667\), and rejects none. These procedures can be more conservative for this set of values, but their arbitrary-dependence guarantee makes the trade-off explicit.
Common Mistakes and Limitations
- Assuming valid margins imply valid familywise control. Uniformity of each null p-value is a marginal statement; the counterexample shows it does not determine the joint rejection probability.
- Calling Hochberg automatically invalid under dependence. That goes too far. Some dependent structures satisfy conditions under which it is valid. The correct conclusion is that control should not be assumed when those conditions are unsupported.
- Treating an observed dependence estimate as proof. Correlations computed from observed data do not establish the null joint distribution required for a guarantee.
- Using the observed pattern to choose a more favorable procedure. The family and the multiplicity strategy should be prespecified; selecting a procedure after seeing results can compromise the intended error-rate interpretation.
Hochberg’s potential gain in power is valuable only when its guarantee is credible for the design. If dependence is uncertain, Holm or Bonferroni offers FWER control under arbitrary dependence, assuming valid component p-values. A more tailored approach may be appropriate when a justified joint model is available, but its assumptions should be stated and checked. No procedure can repair invalid individual p-values or an incorrectly defined family.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For four hypotheses at \(\alpha=0.05\), what are Hochberg’s thresholds for the ordered p-values \(p_{(1)},\ldots,p_{(4)}\)?
- Why does uniformity of each null p-value not establish Hochberg’s FWER control?
- In the three-test construction, what is the probability that a fixed coordinate falls in \((1/60,1/40]\), and why is it exactly the interval width?
- In Case C of the construction, what event produces a Hochberg rejection, and what is its conditional probability?
- If the p-values are dependent and the dependence condition for Hochberg is unsupported, name one procedure with arbitrary-dependence FWER control and state a necessary input condition.
Solutions. (1) The thresholds are \(\alpha/4=0.0125,\ \alpha/3\approx0.01667,\ \alpha/2=0.025,\) and \(\alpha=0.05\). (2) Marginal uniformity specifies each coordinate separately but not the probability of the joint step-up rejection event. (3) The probability is \((1/80)(2/3)=1/120\), equal to the interval width \(1/40-1/60=1/120\). (4) All three Case C values must be at most \(1/20\); its conditional probability is \((1/39)^3\). (5) Holm has arbitrary-dependence FWER control when each component p-value is valid under its null.