One-Sided and Two-Sided Multiplicity Families

Why Direction Is Part of the Family

A multiplicity family is not defined only by a list of treatment comparisons. It also depends on the claims being tested. For a treatment compared with control, a one-sided test asks whether the treatment effect is in a specified direction; a two-sided test asks whether it differs in either direction. Those choices determine the null and alternative hypotheses, the relevant tail probability, and how a per-test threshold is interpreted.

Suppose there are \(m\) treatment hypotheses in a family and the familywise error rate target is \(\alpha\). Equal Bonferroni allocation assigns each hypothesis a per-test level \(\alpha/m\). This allocation controls the chance of at least one false rejection among the true null hypotheses, without requiring independence among the tests. The direction of each test must still be specified correctly: Bonferroni allocation does not repair an unsuitable or data-selected alternative.

Definition: Let \(\theta_j\) denote the treatment effect for comparison \(j\), with positive values defined in advance to favour treatment over control. A one-sided superiority hypothesis is \(H_{0j}:\theta_j\leq 0\) versus \(H_{aj}:\theta_j>0\). A two-sided hypothesis is \(H_{0j}:\theta_j=0\) versus \(H_{aj}:\theta_j\neq 0\). A family consists of the \(m\) hypotheses whose rejections contribute to the same prespecified familywise error rate.

The sign convention is essential. If \(\theta_j\) is instead defined as control minus treatment, then positive values favour control, and the superiority alternative must point toward negative values. State the contrast direction and the clinical meaning of a positive effect before interpreting a test.

For a one-sided superiority test, the null includes effects that are zero or favour control. The boundary \(\theta_j=0\) is used to calibrate the test: under the usual location model, the rejection probability over the composite null is greatest at that boundary. For a two-sided test, departures in either direction count against the null. “One-sided” and “two-sided” describe the alternative being tested, not a choice to make after looking at which direction the observed estimate happens to point.

Bonferroni Allocation With Different Directions

The equal Bonferroni rule compares each valid hypothesis-test p-value \(p_j\) with \(\alpha/m\). A p-value is valid for its null if, whenever that null is true, \(P(p_j\leq u)\leq u\) for every \(u\) from 0 to 1. This condition applies to one-sided and two-sided p-values, provided each is calculated for its stated hypothesis.

\text{Reject }H_{0j}\text{ if }p_j\leq \frac{\alpha}{m},\qquad j=1,\ldots,m.

For a one-sided test, the p-value is the probability, under the null boundary, of a test statistic at least as extreme in the prespecified direction as the observed statistic. For a symmetric test statistic such as a standard normal statistic \(Z\), the corresponding critical value for an upper-tail test is \(z_{1-\alpha/m}\), where \(P(Z>z_{1-\alpha/m})=\alpha/m\) for \(Z\sim N(0,1)\).

For a two-sided test with a symmetric null distribution, the p-value is the probability of a result at least as far from zero in either direction. Comparing it with \(\alpha/m\) puts \(\alpha/(2m)\) in each tail. Thus its positive critical value is \(z_{1-\alpha/(2m)}\). The factor of two is already part of the two-sided test; it is not an additional division of the familywise level beyond the division by \(m\).

Key rule: In an equal Bonferroni family, the per-hypothesis level is \(\alpha/m\), regardless of whether a member is one-sided or two-sided. For a two-sided test, that level is split between its two tails; for a one-sided test, it is all assigned to the prespecified tail.

Here is a justification that does not rely on independence. Let \(I_0\) be the set of indices for which \(H_{0j}\) is true. A familywise error occurs if at least one true null is rejected. The probability of a union of events is no greater than the sum of their probabilities, so

P(\text{at least one false rejection}) \leq \sum_{j\in I_0}P(p_j\leq \alpha/m) \leq \sum_{j\in I_0}\frac{\alpha}{m} =\frac{|I_0|}{m}\alpha \leq \alpha.

The argument uses only validity of each p-value and the union bound. It remains true when test statistics are dependent, as often happens when several treatment arms share a control group. It may be conservative, particularly when the tests are strongly dependent, but it provides a transparent familywise guarantee.

Worked Examples

Worked Example: One-Sided Superiority in Three Comparisons

Hypothetical data. A trial compares three treatments with a shared control on a continuous outcome. For each comparison, define \(\theta_j=\mu_{Tj}-\mu_C\), so positive values favour treatment. The protocol specifies superiority only. Assume independent normally distributed patient outcomes, a common known standard deviation of 10 outcome units, and 100 patients per arm. These assumptions make the difference in sample means normally distributed with standard error \(\sqrt{10^2/100+10^2/100}\). The three tests may be dependent because they share control patients; Bonferroni does not require independence.

There are \(m=3\) one-sided superiority hypotheses and the familywise target is \(\alpha=0.05\). The per-test threshold is

\frac{\alpha}{m}=\frac{0.05}{3}=0.01667.

For one comparison, suppose the observed treatment-minus-control mean difference is 3.2 outcome units. Its standard error is \(\sqrt{100/100+100/100}=\sqrt{2}=1.414\) outcome units, giving \(Z=3.2/1.414=2.263\). The upper-tail p-value is approximately \(P(Z_{\!N}>2.263)=0.0118\), where \(Z_{\!N}\sim N(0,1)\). Since \(0.0118<0.01667\), this comparison rejects its one-sided null at the Bonferroni threshold. The result is evidence of a positive treatment effect under the stated model; it is not evidence that every treatment comparison is effective.

For reference, the one-sided critical value is approximately \(z_{0.98333}=2.128\). The observed statistic, 2.263, exceeds that value, agreeing with the p-value decision.

Worked Example: The Same Result Under a Two-Sided Family

Use the hypothetical comparison and model assumptions in the previous example, but suppose the protocol asks whether each treatment effect differs from zero in either direction. There are still three hypotheses, so the two-sided p-value threshold remains \(0.05/3=0.01667\). For the observed \(Z=2.263\), the two-sided p-value is approximately \(2(0.0118)=0.0236\). Since \(0.0236>0.01667\), this comparison does not reject its two-sided null at the familywise Bonferroni threshold.

The same observed statistic therefore gives different decisions for the two prespecified questions: evidence passes the one-sided superiority threshold but not the two-sided threshold. Equivalently, the two-sided critical value is \(z_{1-0.01667/2}=z_{0.99167}\), approximately 2.394; the observed statistic does not exceed it. One must not report the one-sided conclusion if the actual planned claim was two-sided.

Worked Example: A Mixed Family of One-Sided and Two-Sided Tests

Hypothetical plan. A study prespecifies four hypotheses at \(\alpha=0.05\): two one-sided superiority tests for two treatments on the primary outcome, and two two-sided tests for treatment effects on a safety outcome. Assume each analysis uses a correctly specified model and yields a valid p-value under its own null. The four tests comprise one family because all four rejections contribute to the same error-rate claim.

The equal per-hypothesis threshold is \(0.05/4=0.0125\). Each one-sided primary test is compared with 0.0125 in its designated tail. For a symmetric normal test, its upper-tail critical value is \(z_{0.9875}=2.241\) when testing treatment benefit. Each two-sided safety test is also compared with 0.0125, but its two tails each have probability \(0.0125/2=0.00625\); its critical values are approximately \(-z_{0.99375}=-2.498\) and \(+2.498\).

Suppose a primary test has \(p=0.010\) and a safety test has two-sided \(p=0.011\). Both pass their respective thresholds. A safety test with \(p=0.014\) does not pass, even if its estimate points toward an important clinical concern. The direction affects how its p-value is formed, but the shared family allocation is still based on the four prespecified hypotheses.

Defining the Family and Choosing the Direction

A family should be specified by the scientific claims whose simultaneous false rejection would be problematic. It may include treatment-versus-control contrasts, endpoints, or both, depending on the trial objectives. The count \(m\) is the number of hypotheses in that declared family, not the number of p-values that happen to be small or the number of results selected for reporting. A two-sided hypothesis is one family member, not two, because its two tails are parts of one null-versus-alternative question.

There is a different construction when the scientific claims are themselves directional. For one contrast, the two separate hypotheses “treatment is better” and “treatment is worse” are two directional claims, not one two-sided p-value. If a family explicitly includes both directional claims for each of \(k\) contrasts, it contains \(2k\) hypotheses. Equal Bonferroni allocation then gives \(\alpha/(2k)\) to each directional claim. This is not the same as a two-sided test at level \(\alpha/k\): in a symmetric setting their numerical tail cutoffs can coincide, but the family members and the claims being controlled differ.

The direction must be chosen from the scientific objective, not from the observed estimates. If the investigator examines whether the estimate is positive or negative and then reports the corresponding one-sided p-value, the resulting procedure has used the data to choose the tail. Its p-value is not generally valid at the stated one-sided level. A prespecified two-sided test, or a properly adjusted procedure for the directional claims, is needed when either direction could count as a finding.

Diagnostics, Alternatives, and Limitations

Before applying the threshold, check that each hypothesis has a clear contrast sign, that one-sided alternatives match the intended claims, that all members of the family are listed, and that the p-values are valid for the stated models. For a normal approximation, inspect whether the standard error and reference distribution are appropriate; if the assumed variance or distribution is not justified, the resulting p-value may fail the validity condition needed by the Bonferroni argument. Also check whether all tests use the same analysis population and whether missing data or model choices affect the planned comparisons.

Bonferroni controls multiplicity across the specified family at a given analysis. It does not by itself adjust for repeated interim looks, unplanned endpoints, or data-dependent changes to the family. If an analysis includes multiple stages or repeated opportunities to stop, those sources of multiplicity require an appropriate design-level procedure in addition to the across-hypothesis allocation described here.

Bonferroni is simple and valid under arbitrary dependence, but can be conservative. Earlier material on Holm testing gives a step-down alternative that also controls familywise error and is never less powerful than the simple Bonferroni procedure when applied to the same valid p-values. Closed testing and graphical procedures can exploit logical structure or prespecified alpha transfer, but their hypotheses and rules must be defined in advance. These alternatives do not make post hoc choice of a tail acceptable.

Common mistake: “There are three treatment comparisons, so use \(0.05/3\) for each. For a two-sided test, divide by three again to get \(0.05/9\).” The second division is incorrect. The two-sided p-value is compared with \(0.05/3\); its tail probability is split between the two sides, giving \(0.05/6\) per tail. Conversely, using \(0.05/3\) in each tail would test at a total level of \(0.10/3\), not \(0.05/3\).

The central practical distinction is between the family allocation and the tail allocation. First count the prespecified hypotheses to obtain the Bonferroni threshold. Then use the one-sided or two-sided p-value appropriate to each hypothesis. Keeping those choices separate makes the error-rate claim auditable and the treatment conclusions interpretable.

Check Your Understanding

Answer each question using the stated family and direction. Concise solutions follow the questions.

A family contains five one-sided superiority hypotheses at \(\alpha=0.025\). What is the per-test threshold?
A family contains four two-sided hypotheses at \(\alpha=0.04\). What total threshold is used for each p-value, and what probability is assigned to each tail under a symmetric test?
For a one-sided upper-tail normal test, \(m=2\), \(\alpha=0.05\), and \(Z=2.10\). The p-value is approximately 0.0179. Does it pass the Bonferroni threshold?
An analyst observes a negative estimated effect, then reports a lower-tail one-sided test although the protocol specified upper-tail superiority. Explain the problem.
For each of three contrasts, a trial treats “benefit” and “harm” as separate directional claims in one family. How many hypotheses are in the family, and what is the equal Bonferroni allocation at \(\alpha=0.03\)?

Solutions. (1) The per-test threshold is \(0.025/5=0.005\). (2) Each two-sided p-value is compared with \(0.04/4=0.010\); under a symmetric test, each tail has \(0.010/2=0.005\). (3) The threshold is \(0.05/2=0.025\). Since \(0.0179<0.025\), the test passes the Bonferroni threshold for the specified upper-tail hypothesis. (4) The analyst selected the tail after seeing the data and changed the planned claim. The reported lower-tail p-value does not test the prespecified upper-tail superiority hypothesis at its stated level; the analysis must follow the plan or use a valid procedure accounting for the different directional claims. (5) There are \(3\times2=6\) directional hypotheses. Equal allocation is \(0.03/6=0.005\) per hypothesis.