Why Use a Step-Up Procedure?
A multi-arm trial can produce several p-values for comparisons with a shared control. Testing every comparison at level \(\alpha\) separately does not, in general, keep the probability of one or more false rejections within \(\alpha\). A multiplicity procedure must therefore be applied to the prespecified family of hypotheses, not chosen after seeing which comparisons look promising.
Hochberg’s procedure is a step-up method: it starts with the largest p-value and checks thresholds that become less stringent at higher ranks. Under appropriate dependence conditions, it controls the familywise error rate strongly and can reject more hypotheses than Holm’s step-down procedure. Its advantage comes with an important qualification: unlike Holm, Hochberg is not generally justified under arbitrary dependence.
The procedure uses the same number \(m\) throughout: it is the size of the prespecified family, not the number of hypotheses that happen to look promising. A p-value must be valid for its own null hypothesis, meaning that under that null its probability of being at most \(u\) is no greater than \(u\) for every \(0\le u\le1\). The procedure does not repair invalid p-values or a family chosen after inspecting the results.
Applying the Rule
Worked Example: All Four Comparisons Pass
Suppose a hypothetical trial compares four experimental treatments with one control. The prespecified family has \(m=4\) hypotheses, the target familywise error rate is \(\alpha=0.05\), and the sorted unadjusted p-values are \(0.004,\ 0.012,\ 0.018,\ 0.041\). Assume the p-values are valid and their dependence satisfies the conditions discussed below.
The threshold at rank \(i\) is \(0.05/(4-i+1)\). Thus the rank-specific thresholds are \(0.0125,\ 0.01667,\ 0.0250,\ 0.0500\). The comparisons are \(0.004\le0.0125\), \(0.012\le0.01667\), \(0.018\le0.0250\), and \(0.041\le0.0500\). In particular, rank 4 passes, so the largest passing rank is \(k=4\).
Hochberg’s rule rejects all four hypotheses. These are conclusions about the four treatment-control comparisons in this hypothetical family; they do not establish that every experimental treatment is beneficial or that the effects have any particular size.
Worked Example: A Passing Rank Can Override a Failed Lower-Rank Comparison
For another hypothetical family of \(m=4\) valid p-values, let the sorted values be \(0.014,\ 0.016,\ 0.040,\ 0.090\), with \(\alpha=0.05\). Use the same thresholds: \(0.0125,\ 0.01667,\ 0.0250,\ 0.0500\).
At rank 1, \(0.014>0.0125\), so that comparison fails. At rank 2, \(0.016\le0.01667\), so rank 2 passes. Ranks 3 and 4 fail because \(0.040>0.0250\) and \(0.090>0.0500\). The largest passing rank is \(k=2\), and the procedure rejects the hypotheses at ranks 1 and 2.
The first-ranked p-value does not itself satisfy its rank-1 threshold, but this does not change the step-up decision: once rank 2 passes, Hochberg rejects all hypotheses through rank 2. This feature distinguishes the rule from a step-down procedure that stops at its first failed comparison.
Adjusted P-Values
Hochberg-adjusted p-values give the same decisions across possible choices of \(\alpha\). For the hypothesis at ordered rank \(i\), calculate
The minimum over ranks \(j\ge i\) makes the adjusted p-values nondecreasing with rank, as they should be when the raw p-values are ordered. Reject the hypothesis at rank \(i\) when \(p^{H}_{(i)}\le\alpha\). If raw p-values tie, use their common ordered values; the rejection decision does not depend on how tied hypotheses are listed.
Worked Example: Computing Adjusted Values
Return to the sorted p-values \(0.014,\ 0.016,\ 0.040,\ 0.090\) from the previous example. At rank 4, the adjusted value is \(\min(1,1\times0.090)=0.090\). At rank 3 it is \(\min(0.080,0.090)=0.080\). At rank 2 it is \(\min(3\times0.016,0.080,0.090)=0.048\). At rank 1 it is \(\min(4\times0.014,0.048,0.080,0.090)=0.048\).
In rank order, the adjusted p-values are \(0.048,\ 0.048,\ 0.080,\ 0.090\). At \(\alpha=0.05\), the first two are at most \(0.05\) and the other two are greater than \(0.05\), agreeing with the step-up decision to reject exactly the first two hypotheses. Each adjusted value summarizes the smallest familywise significance level at which that ranked hypothesis would be rejected by this procedure.
Why the Dependence Condition Matters
The error rate of interest is the familywise error rate (FWER): the probability of rejecting at least one hypothesis that is true. Strong control means that this probability is at most \(\alpha\) for every configuration of true and false hypotheses in the family, not only when all hypotheses are true.
A useful justification for Hochberg’s strong control is the Simes inequality. For \(m_0\) true null hypotheses, let \(p^0_{(1)}\le\cdots\le p^0_{(m_0)}\) be their ordered p-values. Under mutual independence of the true-null p-values, or under an appropriate positive-regression-dependence condition on the true-null p-values, the inequality states
This inequality is the key probability bound used here; its detailed development belongs with the Simes test. The link to Hochberg’s rule can be justified directly. Suppose at least one true hypothesis is rejected, and let \(i\) be the largest overall rank of a rejected true hypothesis. Let \(j\) be its rank among the true-null p-values, ordering any tied p-values consistently in both lists. Since the procedure rejected through some passing rank \(k\ge i\), ordering gives \(p^0_{(j)}=p_{(i)}\le p_{(k)}\le\alpha/(m-k+1)\). All ranks from \(i+1\) through \(k\) are rejected and therefore cannot be true. Thus all \(m_0-j\) true hypotheses ranked after \(j\) occur after rank \(k\), so \(m-k\ge m_0-j\) and \(m-k+1\ge m_0-j+1\). Hence
Thus any false rejection by Hochberg implies the event in the Simes inequality for the true-null p-values. Its probability is at most \(\alpha\), establishing strong FWER control under the stated conditions.
For a family of independent tests, independence of the true-null p-values is sufficient. In multi-arm trials, however, treatment-control statistics often share a control group and are correlated. Correlation alone does not establish the needed condition: one must justify independence or suitable positive regression dependence for the p-values under the true nulls. For example, a suitably specified one-sided multivariate normal test model with the required positive dependence may justify the positive-dependence condition. It should not be assumed automatically, especially for two-sided p-values, whose dependence structure may not inherit the required property from positively correlated test statistics.
A Dependence-Aware Application
Worked Example: Shared Control and a Justified Positive-Dependence Model
Consider a hypothetical three-arm trial with two experimental treatments and a shared control, giving \(m=2\) treatment-control hypotheses. Suppose the prespecified tests are one-sided in the same beneficial direction and produce valid p-values \(0.009\) and \(0.031\). Assume the trial’s specified joint test model justifies the required positive regression dependence for the true-null p-values. The dependence assumption is part of this example; it is not inferred merely from the fact that the control is shared.
With \(\alpha=0.05\), the ordered thresholds are \(0.05/2=0.0250\) at rank 1 and \(0.0500\) at rank 2. Since the largest p-value satisfies \(0.031\le0.0500\), rank 2 passes and both hypotheses are rejected.
For the dependence justification, the assumed positive-regression-dependence condition permits the Simes inequality for any subset of true-null p-values. The strong-control argument above then bounds the chance of one or more false rejections by \(0.05\). If the condition had not been justified, this numerical rejection rule would still be computable, but its claimed FWER guarantee would not follow from this argument.
Comparison, Diagnostics, and Limitations
Holm’s step-down procedure controls strong FWER under arbitrary dependence when the individual p-values are valid. Hochberg can be more powerful when its independence or positive-dependence conditions hold: if Holm rejects through rank \(k\), its sequential comparisons include a passing comparison at rank \(k\), so Hochberg’s step-up rule also rejects through at least that rank. The reverse need not hold, as the example with p-values \(0.014\) and \(0.016\) showed. When dependence is unclear or arbitrary, Holm is the safer of these two procedures. Bonferroni is another simple option that controls FWER under arbitrary dependence but can be more conservative.
Before relying on Hochberg’s conclusion, check the following:
- Family: Confirm that the hypotheses form the prespecified family relevant to the error-rate claim. Do not quietly omit inconvenient comparisons.
- Input p-values: Use valid, unadjusted p-values for the individual hypotheses. Do not apply Hochberg to values that have already been adjusted by another multiplicity method.
- Dependence: Identify the argument for independence or positive regression dependence among true-null p-values. A shared-control design makes this check especially important.
- Ordering and arithmetic: Keep the family size fixed at \(m\), sort all \(m\) p-values, and use \(\alpha/(m-i+1)\) at rank \(i\). Check that the largest passing rank, rather than the number of individual comparisons that pass their own thresholds, determines the rejection set.
Hochberg’s procedure controls a familywise error rate, not the false discovery rate, and its adjusted p-values do not measure effect magnitude or clinical importance. Its guarantee also depends on valid marginal tests and the stated joint dependence conditions. If those conditions cannot be supported, do not treat a more favorable result than Holm as evidence that Hochberg remains valid under arbitrary dependence.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For \(m=3\), \(\alpha=0.05\), and ordered p-values \(0.010,\ 0.030,\ 0.070\), which hypotheses does Hochberg reject?
- For \(m=4\), explain why a failed rank-1 comparison does not necessarily prevent rank-1 rejection.
- What dependence conditions justify the strong FWER guarantee described here?
- Why does a shared control not, by itself, prove that Hochberg’s dependence condition holds?
- If the dependence condition cannot be justified, which of Hochberg or Holm offers the stated strong control under arbitrary dependence?
Solutions. (1) The thresholds are \(0.01667,\ 0.0250,\ 0.0500\). Rank 1 passes because \(0.010\le0.01667\), while ranks 2 and 3 fail. The largest passing rank is 1, so Hochberg rejects the hypothesis at rank 1. (2) If a higher rank \(k\) passes, the step-up rule rejects every hypothesis through \(k\), even if a lower-ranked p-value exceeds its own, more stringent threshold. (3) Independence among true-null p-values, or an appropriate positive-regression-dependence condition on them, together with valid individual p-values. (4) Sharing a control induces dependence, but the resulting p-value dependence may not satisfy the condition; the joint test model must support it. (5) Holm controls strong FWER under arbitrary dependence for valid p-values; Hochberg’s guarantee here requires its additional dependence condition.