Multiple Comparison Procedures for Dose-Ranging Studies

A one-sided multiplicity procedure may therefore be appropriate when the protocol and clinical question justify a directional hypothesis.

If clinically important effects can occur in either direction, a two-sided procedure is generally required.

Multiple Comparisons With Unequal Sample Sizes

Dose-ranging studies do not always have equal sample sizes.

For example:

The contrast estimates remain straightforward, but their standard errors depend on the group sizes.

For independent groups with a common variance estimate \(s^2\), the standard error of a dose-versus-placebo difference is approximately:

The multiplicity adjustment must then be based on the appropriate joint distribution of the resulting test statistics.

ANCOVA in Dose-Ranging Trials

Many clinical trials analyze a continuous endpoint using ANCOVA rather than an unadjusted comparison of post-treatment means.

A typical model is:

The treatment coefficients or estimated marginal means can then be used to construct dose-versus-placebo contrasts.

The multiplicity procedure applies to the planned contrasts, not necessarily to a collection of separate unadjusted ANCOVA models.

Multiplicity After ANCOVA

Suppose the adjusted treatment means are:

The clinically relevant contrasts are still:

The difference is that these estimates are adjusted for the covariates included in the ANCOVA model.

Trend Tests

A trend test asks whether the outcome changes systematically with dose.

A simple linear model can be written as:

The trend hypothesis is:

against a directional alternative such as:

A significant trend provides evidence of a dose-response relationship.

However, a linear trend test can miss nonlinear relationships.

Nonlinear Dose-Response Patterns

Real pharmacologic dose-response relationships may be:

• Approximately linear
• Monotonic but nonlinear
• Plateau-shaped
• Inverted U-shaped
• Delayed or threshold-like

Therefore, testing only a linear trend may be too restrictive.

For example, consider:

with effects:

The relationship is clearly increasing but appears to plateau.

A linear model may describe the general pattern but may not represent the plateau particularly well.

Model-Based Dose-Response Analysis

Instead of testing individual doses independently, investigators can fit a dose-response model.

For example, an Emax model is:

where:

• \(E_0\) is the baseline or placebo-level response.
• \(E_{\max}\) is the maximum treatment effect.
• \(ED_{50}\) is the dose producing half of the maximum effect.
• \(D\) is dose.

This approach answers a somewhat different question from Dunnett's test.

Dunnett asks which prespecified doses differ from placebo.

A dose-response model asks: what underlying dose-response relationship best describes the observed data?

Closed Testing and Dose-Ranging Studies

Multiplicity can also be controlled through a closed testing principle.

The basic idea is that a hypothesis can be rejected only when the appropriate intersection hypotheses have also been rejected.

For example, with hypotheses:

a closed family contains intersection hypotheses such as:

Closed testing can provide strong FWER control while allowing structured testing strategies.

In a dose-ranging study, it can be particularly useful when the hypotheses have a logical hierarchy or ordered structure.

Hierarchical Dose Testing

Suppose the protocol defines a testing hierarchy:

Such strategies can preserve strong error control if the hierarchy and testing rules are specified appropriately.

Why Dose Order Matters

Suppose doses are:

An ordered procedure can use the expectation:

This is a stronger structural assumption than simply saying that at least one dose differs from placebo.

When the assumption is reasonable, exploiting it can improve power.

When it is not reasonable, it can produce a poorly targeted analysis.

Bonferroni-Holm as a General Alternative

If the comparison family does not have a natural common-control structure, Holm's step-down procedure is often a useful general-purpose method.

Suppose the unadjusted p-values are:

Sort them:

Holm compares sequentially against:

for \(\alpha=0.05\), these are:

The procedure stops at the first failure.

Holm is generally more powerful than simple Bonferroni while maintaining strong FWER control.

When Should Holm Be Used Instead of Dunnett?

Holm may be attractive when:

• The hypotheses do not all compare doses with a common placebo.
• The comparison family contains heterogeneous contrasts.
• The treatment groups do not have a natural control structure.
• A general-purpose step-down method is preferred.

Dunnett is usually more naturally targeted when the primary family consists specifically of several treatment-versus-common-control comparisons.

Multiple Endpoints and Multiple Doses

Dose-ranging multiplicity becomes more complicated when there are multiple endpoints.

For example, suppose the trial has:

• Four active doses
• One primary endpoint
• One key secondary endpoint

There may already be eight dose-versus-placebo hypotheses.

If both endpoints are confirmatory, the multiplicity strategy should address the combined family appropriately.

Possible strategies include:

• Hierarchical testing
• Gatekeeping
• Bonferroni or Holm adjustment
• Resampling-based methods
• Graphical multiple-testing procedures
• Separate prespecified families when scientifically justified

Multiplicity and Dose Selection

Dose-ranging studies often have a development objective rather than a simple yes/no efficacy objective.

The purpose may be to identify:

• The minimum effective dose
• A dose-response plateau
• A dose with the best benefit-risk profile
• A dose suitable for Phase III

Therefore, the statistical analysis should not reduce the entire study to the question: which dose has the smallest p-value?

A useful dose-selection framework combines:

R Implementation: Dunnett's Test

R provides several ways to perform multiple comparisons.

A simple analysis can begin with a one-way ANOVA:

fit <- aov(
  change ~ dose,
  data = dat
)

summary(fit)

The multcomp package can then be used for general linear-hypothesis testing.

library(multcomp)

fit <- aov(
  change ~ dose,
  data = dat
)

dunnett_fit <- glht(
  fit,
  linfct = mcp(dose = "Dunnett")
)

summary(dunnett_fit)
confint(dunnett_fit)

The output provides simultaneous inference for the treatment-versus-control comparisons.

Specifying the Reference Group

It is important that placebo be the reference level.

dat$dose <- factor(
  dat$dose,
  levels = c(
    "Placebo",
    "5 mg",
    "10 mg",
    "20 mg",
    "40 mg"
  )
)

fit <- aov(
  change ~ dose,
  data = dat
)

This makes the intended treatment structure explicit.

Dunnett With an ANCOVA Model

For a baseline-adjusted continuous endpoint, a linear model might be:

fit <- lm(
  endpoint ~ dose + baseline,
  data = dat
)

library(emmeans)

emm <- emmeans(
  fit,
  ~ dose
)

contrast(
  emm,
  method = "trt.vs.ctrl",
  ref = "Placebo",
  adjust = "dunnett"
)

This approach is often particularly useful in clinical-trial analyses because it separates the modeling of the endpoint from the specification of the multiplicity-adjusted treatment contrasts.

Simultaneous Confidence Intervals in R

contrast(
  emm,
  method = "trt.vs.ctrl",
  ref = "Placebo",
  adjust = "dunnett"
) |>
  confint()

The resulting intervals can be interpreted jointly under the selected multiplicity adjustment.

Williams-Type Contrasts in R

Williams-type analyses require the dose ordering to be represented correctly.

Depending on the exact procedure and analysis model, specialized contrast matrices or dose-response methods may be used.

A general contrast framework can be illustrated using:

emm <- emmeans(
  fit,
  ~ dose
)

contrast(
  emm,
  method = list(
    "High dose vs placebo" =
      c(-1, 0, 0, 0, 1)
  )
)

For an actual Williams procedure, use a validated implementation appropriate to the dose structure rather than manually approximating the procedure with an arbitrary collection of contrasts.

Trend Analysis in R

If dose is scientifically treated as a quantitative variable, a regression model can be used.

fit_linear <- lm(
  change ~ dose,
  data = dat
)

summary(fit_linear)

A significant coefficient for dose provides evidence of a linear trend.

However, dose should not automatically be entered as a numeric variable.

If the dose levels are 5, 10, 20, and 40 mg, treating dose numerically assumes that the numerical distances have meaningful implications for the response model.

For example, the jump from 5 to 10 mg is not necessarily pharmacologically equivalent to the jump from 20 to 40 mg.

Factor Dose Versus Numeric Dose

Choosing among these representations is a scientific modeling decision, not merely a programming decision.

Trend Tests Can Be More Powerful

If the true treatment effects increase monotonically with dose, a trend test can combine information across dose groups.

Suppose the true effects are:

A collection of individual dose-versus-placebo tests may each have only moderate power.

A trend test can instead use the ordered pattern as a whole.

This can be advantageous when the primary scientific hypothesis is about dose-response rather than identification of a particular dose.

But Trend Tests Have Limitations

A significant trend does not necessarily mean that the highest dose is the best clinical dose.

For example, the dose-response relationship might continue upward while adverse events increase substantially.

Likewise, a trend test may be insensitive to a threshold effect or an unusual nonmonotone response.

Therefore: trend evidence and dose selection are related but distinct tasks.

Multiplicity-Adjusted P-Values

Suppose the unadjusted p-values for four dose comparisons are:

An adjusted procedure may produce larger p-values because it accounts for the fact that four related hypotheses were tested.

For example, a generic multiplicity adjustment might yield:

The exact values depend on the procedure and covariance structure.

Multiplicity-Adjusted Confidence Intervals

Suppose the estimated treatment effects are:

The simultaneous intervals show that the 20 mg and 40 mg effects are clearly above zero, while the intervals for the lower doses include zero.

This provides more information than adjusted p-values alone because the magnitude and precision of the treatment effects remain visible.

Multiplicity and Sample-Size Planning

Multiplicity affects power and therefore affects sample-size requirements.

Suppose an unadjusted comparison would require a critical value of approximately 1.96 for a two-sided 5% test.

A multiplicity-adjusted procedure generally requires a more stringent critical value.

Therefore, maintaining the same power may require a larger sample size.

This is one reason multiplicity should be incorporated during study planning rather than added after the trial has been designed.

Power for Dunnett Comparisons

Suppose the investigator wants adequate power to detect a clinically meaningful difference \(\delta\) for one or more doses.

The standardized effect can be represented approximately as:

where \(\sigma\) is the residual standard deviation.

The power calculation must then account for:

• Number of active doses
• Control-group size
• Active-group sizes
• Correlation among treatment-control comparisons
• One-sided versus two-sided testing
• Target family-wise alpha
• Clinically meaningful effect size

Software implementing Dunnett's distribution is generally preferable to trying to approximate the adjustment manually.

Why More Doses Can Reduce Power

Adding more dose groups may seem beneficial because it provides more information about dose-response.

But if the confirmatory family grows, multiplicity becomes more demanding.

For example:

can increase the number of treatment-versus-placebo hypotheses.

Unless the additional dose groups contribute useful information, they may reduce the efficiency of the confirmatory comparisons.

Choosing Dose Levels

Statistical efficiency depends partly on where the doses are placed.

If the doses are too closely clustered, they may provide little additional information.

If they are extremely widely separated, important features of the dose-response curve may be missed.

Dose selection should therefore consider:

• Preclinical pharmacology
• Pharmacokinetics
• Exposure-response evidence
• Expected efficacy range
• Safety margins
• Potential plateau
• Potential maximum tolerated dose

Placebo Is a Shared Comparator

One of the most important structural features of dose-ranging studies is that all active doses commonly share the same placebo group.

This creates correlated treatment-control comparisons.

That correlation is not merely a nuisance. Dunnett's procedure uses it to obtain a more efficient multiplicity adjustment.

If the same placebo observations appear in multiple comparisons, the resulting test statistics are statistically related.

Pairwise All-Dose Comparisons

Suppose the study has five groups.

All pairwise comparisons produce:

comparisons.

But the scientific objective of a dose-ranging study is rarely to establish that every dose differs from every other dose.

The relevant questions are more often:

• Does any dose beat placebo?
• Where does efficacy begin?
• Is there evidence of a plateau?
• Does efficacy increase with exposure?
• Which dose provides an acceptable benefit-risk profile?

Therefore, an all-pairwise Tukey analysis may answer a broader question than the one actually required.

Dunnett Versus Tukey

Common Mistakes

1. Testing every dose versus placebo at 0.05 without adjustment. This can inflate the family-wise type I error.
2. Using Bonferroni automatically. Bonferroni is valid but may be unnecessarily conservative when all comparisons share a common control.
3. Using Dunnett for every dose-response question. Dunnett is excellent for dose-versus-placebo comparisons but does not directly answer every question about dose-response shape.
4. Using a trend test when monotonicity is implausible. A trend procedure relies on an ordered dose-response structure.
5. Treating numeric dose as automatically linear. Dose values such as 5, 10, 20, and 40 mg do not imply a linear response model.
6. Choosing the dose solely because it has the smallest p-value. Clinical dose selection should consider efficacy magnitude, safety, precision, and dose-response shape.
7. Reporting unadjusted confidence intervals as if they were simultaneous. Multiplicity applies to interval inference as well as hypothesis tests.
8. Adding post hoc contrasts after seeing the data. The inferential family should be defined before the confirmatory analysis.
9. Ignoring the covariance among comparisons. Shared placebo groups induce correlation that procedures such as Dunnett explicitly account for.
10. Confusing exploratory dose-response modeling with confirmatory testing. A fitted model may be useful for understanding the dose-response curve, but its inferential role should be prespecified.

A Practical Decision Framework

Recommended Analysis Structure for a Phase II Dose-Ranging Trial

A robust statistical analysis might contain several complementary components.

These analyses answer complementary questions and should not be treated as interchangeable.

Example Statistical Analysis Plan Language

A protocol might state that the primary efficacy analysis will compare each active dose with placebo using a prespecified multiplicity-adjusted procedure.

For example:

The exact wording should reflect the actual protocol, estimand, endpoint, analysis population, and statistical model.

Multiplicity and Missing Data

The multiplicity adjustment does not replace the missing-data strategy.

If the primary endpoint is missing for some patients, the analysis model may require assumptions or methods for handling missing observations.

Possible approaches include:

• Mixed models for repeated measures
• Multiple imputation
• Reference-based imputation
• Pattern-mixture sensitivity analyses
• Other prespecified estimand-aligned methods

The multiplicity procedure should then be applied to the prespecified estimand-relevant treatment contrasts.

Multiplicity and Estimands

Modern clinical-trial analysis begins by defining the treatment effect of interest before selecting the statistical test.

For a dose-ranging study, the estimand may specify:

• Population
• Treatment condition
• Endpoint
• Intercurrent-event strategy
• Summary measure

The multiple-comparison procedure should then be aligned with the resulting treatment comparisons.

When Dunnett Is Usually the Natural Choice

Dunnett is particularly attractive when all of the following are true:

• There is a single common control group.
• There are multiple active doses.
• The primary confirmatory question is dose versus placebo.
• FWER control is required.
• Individual dose comparisons are clinically meaningful.

For this structure, Dunnett directly targets the main inferential problem.

When Williams May Be More Informative

Williams-type procedures can be attractive when:

• Doses have a meaningful natural ordering.
• A monotone dose-response relationship is plausible.
• The scientific question concerns increasing treatment effect with dose.
• The investigator wants to exploit dose ordering for statistical efficiency.

They can be particularly useful when the goal is to establish that a treatment effect emerges and increases across ordered doses rather than to test every dose independently.

When a Dose-Response Model Is Preferable

A model-based approach may be particularly useful when the development program needs estimates such as:

• ED50
• Emax
• Plateau dose
• Target exposure
• Optimal dose region

In these settings, the study may use a combination of confirmatory dose-versus-placebo testing and model-based dose-response characterization.

A Useful Conceptual Distinction

There are three related but different questions:

Dunnett is particularly suited to Question A.

Williams or trend procedures can contribute to Question B.

Question C requires statistical evidence from the efficacy and dose-response analyses together with safety and pharmacologic information.

Summary of Major Procedures

The Most Important Concept

The most important point is that multiplicity should follow the scientific question.

A dose-ranging trial with placebo and four active doses does not automatically require every possible pairwise comparison.

If the primary question is whether any dose beats placebo, Dunnett's procedure is often the most natural choice.

If the primary question is whether efficacy increases with dose, an ordered procedure such as Williams or a prespecified trend analysis may be more appropriate.

If the objective is to understand the complete dose-response relationship, model-based approaches may provide substantially more useful information than a collection of pairwise p-values.

References

Dunnett, C.W. (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association, 50, 1096–1121.
Dunnett, C.W. (1964). New tables for multiple comparisons with a control. Biometrics, 20, 482–491.
Williams, D.A. (1971). A test for differences between treatment means when several dose levels are compared with a zero dose control. Biometrics, 27, 103–117.
Williams, D.A. (1972). The comparison of several dose levels with a zero dose control. Biometrics, 28, 519–531.
Hsu, J.C. (1996). Multiple Comparisons: Theory and Methods. Chapman & Hall/CRC.
Bretz, F., Hothorn, T. & Westfall, P. (2010). Multiple Comparisons Using R. CRC Press.
Pinheiro, J., Bornkamp, B. & Bretz, F. (2009). Design and analysis of dose-finding studies with continuous responses. Statistics in Medicine.
FDA. Multiple Endpoints in Clinical Trials: Guidance for Industry. U.S. Food and Drug Administration.
ICH E9. Statistical Principles for Clinical Trials. International Council for Harmonisation.