Hochberg and Hommel Procedures

Introduction

Clinical trials frequently evaluate more than one statistical hypothesis. Examples include multiple treatment groups, multiple doses, multiple endpoints, multiple time points, or several prespecified comparisons.

The problem is that testing several hypotheses at the usual 5% significance level increases the probability of making at least one false-positive declaration.

If the investigator wants to maintain strong control of the family-wise error rate (FWER), a multiplicity adjustment is required.

Two important procedures for FWER control are the Hochberg procedure and the Hommel procedure.

The Multiple-Testing Problem

Suppose a clinical trial tests \(m\) null hypotheses:

Let the corresponding raw p-values be:

If every hypothesis is tested independently at level 0.05, the probability of obtaining at least one false-positive result can exceed 5%.

Under independence, if all \(m\) null hypotheses are true, the probability of making no false rejection is approximately:

Therefore the probability of at least one false rejection is:

For five independent tests at \(\alpha=0.05\):

So the chance of at least one false-positive result is approximately 22.6% if all five null hypotheses are true and each test is performed independently at 5%.

What Is Family-Wise Error Rate?

The FWER is defined as the probability of rejecting at least one true null hypothesis.

A procedure strongly controls the FWER at level \(\alpha\) if the FWER is no greater than \(\alpha\) for every configuration of true and false null hypotheses within the procedure's stated assumptions.

In confirmatory clinical trials, strong FWER control is often important when multiple efficacy claims could otherwise lead to a positive overall conclusion.

Bonferroni: The Starting Point

The Bonferroni procedure is one of the simplest methods for FWER control.

For \(m\) hypotheses, reject \(H_i\) when:

For five hypotheses at an overall one-sided significance level of 5%:

Every individual p-value must therefore be at most 0.01.

Bonferroni is attractive because its validity does not require independence among the tests.

However, it can be conservative.

Holm's Step-Down Procedure

Before understanding Hochberg, it helps to understand Holm's procedure.

Sort the p-values:

Holm compares the smallest p-value with:

the second-smallest with:

and so on.

The procedure moves from the smallest p-value toward the largest and stops when the first hypothesis fails its criterion.

This is called a step-down procedure.

Hochberg Takes the Opposite Direction

Hochberg uses the same ordered p-values:

but examines them from the largest p-value toward the smallest.

This makes Hochberg a step-up procedure.

The Hochberg Critical Values

For \(m\) hypotheses, the ordered p-values are compared with:

More precisely, for ordered p-value \(p_{(i)}\), the corresponding Hochberg critical value is:

Thus:

Hochberg's Decision Rule

Let \(j\) be the largest index satisfying:

Then reject:

If no such \(j\) exists, reject none of the hypotheses.

Hochberg Worked Example

Suppose a clinical trial has five confirmatory hypotheses with raw p-values:

Let:

The ordered p-values are:

Now construct the Hochberg comparison table.

The first qualifying value encountered while moving downward is:

Therefore Hochberg rejects the first two ordered hypotheses:

In the original labeling, these correspond to \(H_1\) and \(H_2\).

Why Hochberg Can Reject More Than Bonferroni

Bonferroni uses the same threshold for every hypothesis:

Only the p-value 0.003 would therefore be significant by Bonferroni.

Hochberg, however, recognizes that the second-smallest p-value satisfies:

and consequently rejects both of the two smallest hypotheses.

This is an example of the power advantage created by the step-up structure.

Hochberg Versus Holm

Both procedures use ordered p-values, but they proceed in opposite directions.

Dependence Assumptions Matter

The increased power of Hochberg is not free.

Unlike Holm's procedure, which controls the FWER under arbitrary dependence among p-values, the classical Hochberg procedure relies on assumptions about the dependence structure.

In particular, Hochberg's procedure has strong FWER control under independence and under important classes of positive dependence, including the standard positive regression dependence framework commonly used for multiple-testing theory.

Therefore the investigator should not simply select Hochberg because it gives smaller adjusted p-values.

Enter the Hommel Procedure

The Hommel procedure is more sophisticated.

It is based on the closed-testing principle combined with Simes' local test.

The procedure can be viewed as a way of testing every relevant intersection hypothesis while exploiting the Simes test to obtain greater power than simple Bonferroni-based closed testing.

Hommel is computationally more complicated than Hochberg, but modern software makes its implementation straightforward.

The Closed-Testing Principle

Suppose there are hypotheses:

The closed-testing principle considers intersection hypotheses such as:

and the global intersection:

An individual hypothesis can be rejected only if all relevant intersection hypotheses containing it are rejected by their corresponding local tests.

Why Closed Testing Controls FWER

Suppose at least one null hypothesis is true.

For a false rejection of that true null hypothesis to occur, every relevant intersection hypothesis containing it must also have been rejected.

Because the global intersection and local tests are constructed at level \(\alpha\), the closed-testing procedure provides strong control of the FWER.

This is the fundamental theoretical framework behind many powerful multiplicity procedures.

Simes' Test

The Hommel procedure uses Simes' inequality and test.

For \(k\) p-values ordered as:

the Simes test rejects the global null at level \(\alpha\) if:

for at least one \(i\).

Equivalently, the Simes p-value can be written as:

with the result truncated at 1 if necessary.

Hommel as a Closed Simes Procedure

The Hommel procedure applies Simes-type tests to the intersection hypotheses created by the closed-testing framework.

Conceptually, for every possible subset of hypotheses, Hommel asks whether the corresponding intersection null can be rejected by a Simes test.

The resulting individual decisions are then obtained through the closed-testing logic.

This is why Hommel is often described as a closed-testing procedure based on Simes tests.

Hommel Is More Than a Simple Step-Up Rule

Hochberg can be implemented using a relatively simple ordered comparison.

Hommel requires considering a collection of possible subset sizes.

For \(m\) hypotheses, define:

when such a \(j\) exists.

The Hommel rejection rule can then be expressed using this value and the corresponding adjusted p-values.

In practice, it is much easier and safer to use a validated implementation, such as R's p.adjust(), rather than manually coding the algorithm for routine analysis.

A Conceptual Hommel Example

Suppose five hypotheses have ordered p-values:

The Hommel procedure examines Simes-type evidence across different subset sizes rather than simply comparing each p-value against one fixed threshold.

For the complete family, the Simes criterion is:

Several of the ordered p-values satisfy their corresponding Simes inequalities. This provides evidence against the complete intersection null.

The full Hommel procedure then examines the relevant intersection structure to determine which individual hypotheses can be rejected while preserving strong FWER control.

Hochberg Versus Hommel

Hommel and Hochberg Are Closely Related

The two procedures are not unrelated competing ideas.

Both exploit information in the ordered p-values.

Hochberg can be understood as a particularly efficient step-up procedure based on intersection testing ideas.

Hommel goes further by fully exploiting the closed-testing framework with Simes local tests.

Under the standard dependence conditions under which both procedures are valid, Hommel is at least as powerful as Hochberg and can sometimes reject additional hypotheses.

Adjusted P-Values

Instead of reporting only reject/not-reject decisions, multiplicity procedures can also produce adjusted p-values.

An adjusted p-value can be interpreted as the smallest family-wise significance level at which the corresponding hypothesis would be rejected by that procedure.

For Bonferroni:

For Holm, Hochberg, and Hommel, the adjusted p-values depend on the ordered set of p-values and the specific procedure.

Hochberg Adjusted P-Values

Let:

The raw Hochberg multipliers progress in the opposite direction from Holm's step-down testing sequence.

The unadjusted candidate values are:

and the final adjusted values must be made monotone across the ordered p-values.

In practice, software should be used to avoid implementation errors.

Why Adjusted P-Values Are Useful

Suppose a clinical study reports:

At an overall 5% FWER threshold, endpoints whose adjusted p-values are below 0.05 would be considered statistically significant.

The exact adjusted values depend on the complete set of raw p-values and the procedure used.

R Implementation: Hochberg

R provides Hochberg adjustment directly through p.adjust().

p <- c(
  0.003,
  0.011,
  0.018,
  0.029,
  0.120
)

p.adjust(
  p,
  method = "hochberg"
)

The resulting vector contains the Hochberg-adjusted p-values in the original order of the hypotheses.

R Implementation: Hommel

Hommel is implemented in R using:

p.adjust(
  p,
  method = "hommel"
)

For example:

p <- c(
  H1 = 0.003,
  H2 = 0.011,
  H3 = 0.018,
  H4 = 0.029,
  H5 = 0.120
)

p_hommel <- p.adjust(
  p,
  method = "hommel"
)

p_hommel

Because the original names are preserved, the resulting adjusted p-values can be directly matched to the corresponding hypotheses.

Comparing Several Procedures in R

It is often useful to examine several FWER procedures side by side.

p <- c(
  H1 = 0.003,
  H2 = 0.011,
  H3 = 0.018,
  H4 = 0.029,
  H5 = 0.120
)

data.frame(
  raw = p,
  bonferroni = p.adjust(p, "bonferroni"),
  holm = p.adjust(p, "holm"),
  hochberg = p.adjust(p, "hochberg"),
  hommel = p.adjust(p, "hommel")
)

This is an excellent way to understand how the choice of multiplicity procedure affects the resulting adjusted p-values.

Creating a Decision Table in R

alpha <- 0.05

results <- data.frame(
  hypothesis = names(p),
  raw_p = p,
  hochberg_p = p.adjust(p, "hochberg"),
  hommel_p = p.adjust(p, "hommel")
)

results$hochberg_significant <-
  results$hochberg_p <= alpha

results$hommel_significant <-
  results$hommel_p <= alpha

results

This creates a convenient analysis table containing both adjusted p-values and the corresponding statistical decisions.

A Manual Hochberg Algorithm in R

For teaching and validation purposes, the Hochberg decision algorithm can be implemented explicitly.

hochberg_test <- function(p, alpha = 0.05) {

  m <- length(p)

  ord <- order(p)

  p_sorted <- p[ord]

  j_found <- NA_integer_

  for (j in m:1) {

    critical <- alpha / (m - j + 1)

    if (p_sorted[j] <= critical) {
      j_found <- j
      break
    }
  }

  reject <- rep(FALSE, m)

  if (!is.na(j_found)) {
    reject[ord[1:j_found]] <- TRUE
  }

  data.frame(
    p_value = p,
    reject = reject
  )
}

hochberg_test(p)

This code is useful for understanding the algorithm.

A More Detailed Hochberg Example

Consider six hypotheses with raw p-values:

Let:

The Hochberg thresholds are:

The third ordered p-value satisfies:

Therefore the first three hypotheses are rejected.

Notice how the procedure does not require the second p-value to satisfy \(\alpha/m\).

Instead, once the third ordered p-value establishes the step-up criterion, all smaller p-values are rejected.

Why the Step-Up Direction Creates Power

Suppose several p-values are small but none is extraordinarily small.

Bonferroni may reject only the smallest p-value because it uses:

Hochberg asks a different question: Can a sufficiently large collection of the smallest p-values jointly provide enough evidence to reject several hypotheses while maintaining FWER control?

The step-up structure allows the answer to be yes when the p-values are sufficiently concentrated near zero.

Hochberg Versus Bonferroni

Hochberg Versus Holm

Both are sequential procedures, but the direction is reversed.

The practical consequence is that Hochberg can reject more hypotheses in some situations, but the stronger power claim comes with additional dependence assumptions.

Hochberg Versus Hommel

Hommel is generally more computationally demanding because it exploits the full closed-testing structure.

The relationship can be summarized as:

Under the standard conditions supporting both procedures, Hommel can be at least as powerful as Hochberg.

In other words, there are situations where:

but Hommel will not necessarily produce dramatically different results for every set of p-values.

When the Procedures Give the Same Answer

Multiplicity procedures often produce identical decisions when the evidence is very strong.

For example, if several p-values are extremely small, Bonferroni, Holm, Hochberg, and Hommel may all reject the same hypotheses.

The differences become more important when p-values are near the significance boundary.

Hochberg and Hommel in Clinical Trials

Multiple-testing procedures are particularly important in clinical trials because confirmatory claims may involve:

• Multiple dose levels
• Multiple treatment comparisons
• Multiple primary endpoints
• Primary and key secondary endpoints
• Multiple disease subgroups
• Multiple estimands or analysis strategies
• Hierarchical testing families
• Multiple endpoints within a gatekeeping strategy

The appropriate procedure depends on the structure of the hypotheses, the clinical interpretation of success, and the desired error-control strategy.

Multiplicity Is Not Only About the Number of P-Values

A common mistake is to define a multiplicity problem simply by counting every p-value appearing in a statistical report.

That is not necessarily correct.

The relevant family is determined by the hypotheses that contribute to the confirmatory decision and by the prespecified multiplicity strategy.

For example, exploratory analyses may be reported without being part of the confirmatory FWER-controlled family.

Strong Versus Weak FWER Control

A procedure has weak FWER control if it controls the FWER only when all null hypotheses are true.

A procedure has strong FWER control if it controls the FWER under every configuration of true and false null hypotheses.

Clinical-trial multiplicity procedures generally require strong control when individual claims may be made regardless of which other hypotheses are true.

One-Sided Versus Two-Sided Testing

The Hochberg and Hommel procedures can be applied to either one-sided or two-sided p-values.

The multiplicity procedure itself does not determine whether the underlying hypothesis tests are one-sided or two-sided.

That decision should be made from the scientific question and statistical analysis plan.

For example, if the overall two-sided family-wise level is 5%, the relevant two-sided p-values are adjusted at the family level of:

The use of a two-sided test should not be confused with simply doubling an already adjusted p-value.

Hochberg Is Not the Same as Benjamini-Hochberg

The names can cause confusion.

The Hochberg procedure controls the family-wise error rate under its appropriate dependence conditions.

The Benjamini-Hochberg procedure, in contrast, is designed to control the false discovery rate (FDR).

Adjusted P-Values Versus Critical Values

There are two equivalent ways to present many multiplicity procedures.

Critical-Value Approach

Compare the raw p-values against procedure-specific thresholds.

Adjusted-P-Value Approach

Calculate adjusted p-values and compare each with the common threshold \(\alpha\).

For reporting, adjusted p-values are often convenient because a single column can be added to a statistical table.

Example of a Clinical Results Table

The exact adjusted p-values must come from the specified multiplicity procedure and the complete family of hypotheses.

Multiplicity Families Should Be Prespecified

Suppose a trial has:

• One primary endpoint
• Three key secondary endpoints
• Ten exploratory endpoints

It does not automatically follow that all 14 p-values must be included in one FWER family.

The protocol and SAP may define the confirmatory family differently, provided the strategy is scientifically justified and prespecified.

For example, the primary endpoint may be tested separately while the three key secondary endpoints form a separate multiplicity family.

Hierarchical Testing Can Be Different

A trial may have a logical testing hierarchy such as:

This type of gatekeeping strategy can use the available alpha in a structured way and may be preferable to applying Hochberg or Hommel blindly to every endpoint.

Common Mistakes With Hochberg

1. Applying Hochberg without considering dependence. The procedure's strong FWER validity depends on appropriate assumptions.
2. Calling Hochberg a step-down procedure. Hochberg is step-up; Holm is step-down.
3. Starting from the smallest p-value. The Hochberg algorithm starts at the largest ordered p-value.
4. Rejecting only the p-value that crosses its threshold. Once the qualifying Hochberg index is found, all smaller ordered p-values are rejected.
5. Confusing Hochberg with Benjamini-Hochberg. They control different error criteria.
6. Choosing the method after seeing the results. The multiplicity strategy should be prespecified.
7. Ignoring the definition of the hypothesis family. Not every exploratory p-value necessarily belongs to the same confirmatory family.

Common Mistakes With Hommel

1. Thinking Hommel is just another name for Hochberg. Hommel uses the closed-testing principle with Simes local tests.
2. Manually implementing Hommel without validation. The algorithm is more complicated than Bonferroni or Holm.
3. Using the global Simes test as though it were Hommel. Rejecting the global intersection does not by itself determine all individual hypothesis rejections.
4. Ignoring dependence assumptions behind Simes-based procedures. Validity depends on the conditions under which the Simes local tests control their type I error.
5. Reporting only raw p-values. Confirmatory results should make the multiplicity adjustment explicit.

Comparison of Four Classical FWER Procedures

When Bonferroni May Still Be Preferable

More powerful does not always mean better.

Bonferroni may be attractive when:

• The dependence structure is uncertain.
• The family is relatively small.
• Transparency and simplicity are priorities.
• A conservative procedure is acceptable.
• The analysis needs to be easy to reproduce independently.

The loss of power may be acceptable in exchange for very simple and robust error control.

When Holm May Be Preferable

Holm is often an excellent default when strong FWER control under arbitrary dependence is needed.

It improves on Bonferroni without requiring the additional dependence assumptions associated with Hochberg.

Therefore, if there is no compelling reason to use Hochberg and the dependence structure is not suitable for it, Holm may be the safer choice.

When Hochberg May Be Attractive

Hochberg may be attractive when:

• The p-values are expected to be positively dependent or independent.
• Power is important.
• The testing family is clearly defined.
• A step-up procedure is compatible with the statistical strategy.
• The dependence assumptions can be justified.

When Hommel May Be Attractive

Hommel may be attractive when:

• Strong FWER control is required.
• The Simes test is valid for the dependence structure.
• Additional power over Hochberg is valuable.
• The analysis can use validated software.
• The full closed-testing interpretation is appropriate.

Computational Complexity

Bonferroni and Holm are computationally simple.

Hochberg is also straightforward because it only requires ordering the p-values and examining the step-up thresholds.

Hommel is more computationally involved because its logic effectively considers multiple intersection subsets.

For a modest number of hypotheses, this is not a practical problem.

For large-scale exploratory testing with thousands of hypotheses, however, other methods such as FDR procedures may be more appropriate depending on the scientific objective.

Hochberg and Hommel Are FWER Procedures

The central goal of both procedures is:

under the assumptions required for the particular procedure.

This is fundamentally different from controlling the expected proportion of false discoveries among rejected hypotheses.

That distinction is important when deciding between FWER and FDR methods.

Relationship to Simes' Inequality

Simes' inequality is a key theoretical ingredient in the development of powerful multiple-testing methods.

For valid p-values under the relevant dependence conditions:

This provides the basis for using the Simes test as a valid level-\(\alpha\) test of an intersection null.

Hommel exploits this property throughout the closed-testing framework.

Why Hommel Can Improve on Hochberg

Hochberg uses a specific step-up rejection rule.

Hommel evaluates more of the intersection structure.

Therefore, when the p-values contain patterns of evidence that are not fully captured by the simpler Hochberg rule, Hommel can identify additional valid rejections.

This extra power comes from using more information rather than from relaxing the FWER requirement.

Practical Workflow for Choosing a Procedure

Example SAP Language

A statistical analysis plan might state that a family of confirmatory hypotheses will be tested using a prespecified FWER procedure.

The exact wording should reflect the actual trial design, but the statistical documentation should identify:

• The hypotheses included in the family
• The overall type I error level
• Whether tests are one-sided or two-sided
• The multiplicity procedure
• The ordering or testing structure
• The assumptions supporting the procedure
• The method used to calculate adjusted p-values
• The software and version used for implementation

Reporting in a Clinical Study Report

A clinical study report should make the multiplicity strategy transparent.

For example, a results table might contain:

The exact interpretation should also distinguish statistical significance from clinical importance.

Statistical Significance Is Not Clinical Importance

A multiplicity procedure answers an error-control question:

How much evidence is required before making one or more statistical claims while controlling the probability of a false claim?

It does not answer:

Is the treatment effect clinically meaningful?

A statistically significant result can have a small effect size, while a clinically important effect may fail to reach statistical significance because of limited precision or sample size.

Hochberg and Hommel in a Regulatory Setting

In confirmatory clinical development, multiplicity procedures should be integrated with the overall testing strategy.

The statistical analysis plan should make clear:

• Which hypotheses support formal efficacy claims
• Which hypotheses are exploratory
• How the family-wise error rate is controlled
• How alpha is allocated
• Whether hypotheses are tested concurrently or sequentially
• How multiplicity interacts with estimands and analysis populations

A powerful multiplicity procedure cannot repair an unclear or clinically unjustified hypothesis family.

Validation and Reproducibility

For important confirmatory analyses, the multiplicity calculation should be reproducible.

A useful validation strategy is to compare:

• Software-generated adjusted p-values
• Independent calculations
• Known reference examples
• Alternative validated statistical software

The analysis should also preserve the original hypothesis labels so that sorting the p-values does not accidentally change the identity of the endpoints.

R Example: Preserve Hypothesis Labels

p <- c(
  Primary = 0.008,
  Secondary_A = 0.014,
  Secondary_B = 0.041,
  Secondary_C = 0.003
)

results <- data.frame(
  hypothesis = names(p),
  raw_p = as.numeric(p),
  hochberg = p.adjust(p, method = "hochberg"),
  hommel = p.adjust(p, method = "hommel")
)

results

The output preserves the original hypothesis names while adding the multiplicity-adjusted p-values.

Decision Rules in R

alpha <- 0.05

results$hochberg_reject <-
  results$hochberg <= alpha

results$hommel_reject <-
  results$hommel <= alpha

results

This provides a simple reproducible framework for determining which hypotheses are statistically significant after multiplicity adjustment.

A Compact Mental Model

A useful way to remember the procedures is:

One-Sentence Definitions

What Happens If No Hochberg Threshold Is Crossed?

Suppose:

for every \(i\).

Then Hochberg rejects no hypotheses.

This is an important point. A non-significant largest p-value does not itself imply that the entire family fails. The procedure continues downward until it finds a qualifying p-value.

What Happens If the Largest P-Value Is Below Alpha?

Suppose:

Then Hochberg immediately rejects all hypotheses.

For example, if all five p-values are below 0.05, then the largest one is also below 0.05 and the step-up procedure rejects the entire family.

This makes intuitive sense: if even the largest p-value is below the overall alpha level, all of the smaller p-values provide at least as much evidence.

Why Sorting Does Not Change the Hypotheses

Sorting is only a computational device.

Suppose:

The ordered p-values are:

But the corresponding hypotheses are:

Any rejection decision must therefore be mapped back to the original hypothesis labels.

Hochberg and Hommel Versus Unadjusted Testing

Common Misconception: "More Powerful Means Less Conservative"

It is tempting to think that a more powerful procedure must provide weaker error control.

That is not necessarily true.

Hochberg and Hommel can gain power by using more information about the joint pattern of p-values while maintaining the same nominal FWER under their validity conditions.

The critical issue is not whether the procedure is powerful, but whether its mathematical assumptions are appropriate for the testing problem.

Common Misconception: "Hommel Always Beats Everything"

Hommel can be more powerful than Hochberg under the standard conditions under which both are valid, but that does not mean it is automatically the best procedure for every clinical trial.

The best method depends on:

• The scientific hypotheses
• The family definition
• The dependence structure
• The desired error criterion
• The testing hierarchy
• The regulatory strategy
• The interpretability of the procedure

Common Misconception: "The P-Value Is Adjusted Before Testing"

Conceptually, a multiplicity procedure can be expressed either as adjusted p-values or as critical values.

The underlying statistical evidence has not changed.

Rather, the procedure changes the decision rule so that the probability of false rejection across the entire family is controlled.

A Practical Decision Tree

The Most Important Difference Between Hochberg and Hommel

The easiest way to distinguish the two is:

versus:

Hochberg uses an efficient ordered rejection rule.

Hommel uses a broader closed-testing construction and can therefore exploit additional logical information about intersection hypotheses.

Summary Table

Final Takeaway

The Hochberg and Hommel procedures are important tools for multiple hypothesis testing when the goal is to control the family-wise error rate without relying on the very conservative single-step Bonferroni threshold.

Hochberg is a step-up procedure. It sorts the p-values from smallest to largest, starts with the largest p-value, and searches downward for the first p-value that satisfies its corresponding critical value.

Hommel is more deeply connected to the theory of closed testing. It uses Simes tests for intersection hypotheses and can exploit more of the logical structure of the multiple-testing problem.

Under the standard conditions supporting both procedures, Hommel can be at least as powerful as Hochberg, while Hochberg can be more powerful than Bonferroni and Holm.

References

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802.
Hommel, G. (1983). Tests of the overall hypothesis for arbitrary dependence structures. Biometrical Journal, 25, 423–430.
Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73(3), 751–754.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
Goeman, J.J. & Solari, A. (2014). Multiple testing for exploratory research. Statistical Science, 29(4), 584–597.
Westfall, P.H. & Young, S.S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. Wiley.