Why Logical Constraints Can Improve Step-Down Testing
When several treatment arms are compared pairwise, the null hypotheses are not always logically independent. If treatment arms A and B have the same value of the parameter of interest, and B and C have the same value, then A and C must have the same value as well. This transitivity means that some combinations of pairwise null hypotheses cannot be true together. Shaffer’s procedure uses these restrictions to choose less conservative step-down thresholds than Holm’s procedure can use.
The gain is not based on assuming that test statistics or p-values are independent. It comes from ruling out certain counts of true null hypotheses. The procedure is appropriate only when the hypotheses have the logical structure used to derive those counts and the individual p-values are valid for their corresponding null hypotheses.
This definition assumes that each comparison concerns the same well-defined parameter across arms. For example, \(\theta_a\) might be the mean outcome for arm \(a\) at a prespecified analysis time. Equality of these parameters is transitive, so the true pairwise hypotheses correspond to all pairs within each group of arms having equal parameter values. A hypothesis family built from different estimands, or from relations that are not transitive, does not automatically have this structure.
Constructing Shaffer’s Step-Down Rule
Write the family of \(m\) p-values in increasing order as \(p_{(1)}\leq\cdots\leq p_{(m)}\). Let \(S\) be the set of attainable numbers of true null hypotheses under the logical constraints. At step \(i\), there are \(m-i+1\) hypotheses still in the ordered list. Shaffer’s denominator \(t_i\) is the largest attainable true-hypothesis count no greater than that number.
Starting with \(i=1\), compare \(p_{(i)}\) with \(\alpha/t_i\). Reject \(H_{(i)}\) if \(p_{(i)}\leq\alpha/t_i\), then continue to the next step. At the first step where the inequality fails, stop: that hypothesis and all hypotheses later in the ordered list are not rejected by this procedure. The denominators are determined by the family’s logical structure, not by the observed p-values.
Worked Example: Deriving Denominators for Four Arms
Suppose four hypothetical treatment arms have parameters \(\theta_1,\theta_2,\theta_3,\theta_4\), and the family contains all six pairwise equality hypotheses. Find the possible numbers of true hypotheses and the Shaffer denominators.
If all four parameters are equal, all six pairwise null hypotheses are true. If the equal-parameter groups have sizes \(3+1\), there are \(\binom{3}{2}=3\) true hypotheses. A \(2+2\) grouping gives \(1+1=2\), and a \(2+1+1\) grouping gives \(1\). If all four parameters differ, there are none. Thus \(S=\{6,3,2,1,0\}\).
There are six hypotheses in total. The numbers of hypotheses remaining at successive steps are \(6,5,4,3,2,1\). The largest members of \(S\) no greater than those numbers give the denominators \(6,3,3,3,2,1\), respectively. In particular, the denominator at step two is 3, not 5: five hypotheses remain, but five true pairwise equalities are logically impossible for four arms.
Applying the Procedure
The following example uses the four-arm family just described. The p-values are hypothetical, two-sided p-values for the six pairwise equality tests. Assume that each is valid under its corresponding true null hypothesis, and set the familywise significance level to \(\alpha=0.05\).
Worked Example: Testing Six Pairwise Comparisons
Suppose the sorted p-values are \(0.004,\ 0.014,\ 0.016,\ 0.020,\ 0.210,\ 0.500\). Compare them in order with \(\alpha/t_i\), using \(t_i=(6,3,3,3,2,1)\).
At step 1, the critical value is \(0.05/6=0.008333\). Since \(0.004\leq0.008333\), reject the first hypothesis. At step 2, the critical value is \(0.05/3=0.01667\), and \(0.014\leq0.01667\), so reject the second. At step 3 the critical value remains \(0.01667\); since \(0.016\leq0.01667\), reject the third.
At step 4 the critical value is again \(0.01667\), but \(0.020>0.01667\). Stop at this first failure. The procedure rejects the three hypotheses corresponding to the smallest p-values and does not reject the remaining three. These rejections control the probability of one or more false rejections across the six-comparison family at no more than 0.05, under the stated assumptions.
For comparison, Holm’s procedure would use denominators \(6,5,4,\ldots\). Its first threshold is also \(0.05/6=0.008333\), but its second is \(0.05/5=0.010\). The second p-value, \(0.014\), fails that threshold, so Holm stops after one rejection. Shaffer’s rule can proceed further because the logical structure rules out five true pairwise nulls.
Worked Example: Three Arms and the Final Steps
For three hypothetical arms, there are three pairwise null hypotheses. The possible numbers of true hypotheses are 3 if all parameters are equal, 1 if exactly one pair is equal, and 0 if all parameters differ. Thus \(S=\{3,1,0\}\), and the denominators at the three steps are \(3,1,1\). Take \(\alpha=0.05\) and hypothetical sorted p-values \(0.012,\ 0.028,\ 0.049\).
The first critical value is \(0.05/3=0.01667\), so \(0.012\) passes. The second and third critical values are each \(0.05/1=0.05\); both \(0.028\) and \(0.049\) pass. The procedure therefore rejects all three hypotheses. Holm would compare the second p-value with \(0.05/2=0.025\), which \(0.028\) exceeds, and would stop after the first rejection. The result illustrates how a possible true-hypothesis count of 1 permits a larger threshold at later steps.
Worked Example: Ten Comparisons Among Five Arms
For five hypothetical arms there are \(\binom{5}{2}=10\) pairwise hypotheses. The possible group-size partitions yield attainable true-hypothesis counts \(10\) (sizes \(5\)), \(6\) (\(4+1\)), \(4\) (\(3+2\)), \(3\) (\(3+1+1\)), \(2\) (\(2+2+1\)), \(1\) (\(2+1+1+1\)), and \(0\) (all singletons). Thus \(S=\{10,6,4,3,2,1,0\}\). The resulting denominators are \(10,6,6,4,4,3,2,2,1,1\).
Suppose the sorted hypothetical p-values are \(0.001,\ 0.006,\ 0.008,\ 0.011,\ 0.012,\ 0.015,\ 0.020,\ 0.024,\ 0.040,\ 0.049\), with \(\alpha=0.05\). The first six thresholds are \(0.005,\ 0.008333,\ 0.008333,\ 0.01250,\ 0.01250,\ 0.01667\), and each corresponding p-value is no larger than its threshold. The next two thresholds are \(0.025\), and \(0.020\leq0.025\) and \(0.024\leq0.025\). The last two thresholds are \(0.05\), which \(0.040\) and \(0.049\) also meet. Shaffer’s procedure rejects all ten hypotheses for these hypothetical data.
Why the Procedure Controls Strong Familywise Error
Strong familywise error control means that the probability of rejecting at least one true null hypothesis is at most \(\alpha\), for every configuration of true and false hypotheses in the family. The key step in the justification is to consider the first true hypothesis that the step-down procedure rejects.
Let \(r>0\) be the number of true hypotheses. By assumption, \(r\in S\).
If it occurs at step \(i\), all earlier rejected hypotheses are false. Therefore all \(r\) true hypotheses are among the \(m-i+1\) hypotheses from step \(i\) onward. Hence \(r\leq t_i\).
Because all hypotheses before step \(i\) are false, \(p_{(i)}\) is the smallest p-value among the true hypotheses. A rejection at step \(i\) implies \(p_{(i)}\leq\alpha/t_i\leq\alpha/r\).
For each true null, its valid p-value has probability at most \(\alpha/r\) of being no greater than \(\alpha/r\). The probability that any of the \(r\) true p-values is this small is at most \(r(\alpha/r)=\alpha\).
Thus the probability of any false rejection is at most \(\alpha\). This argument requires valid p-values for the true hypotheses and the stated logical constraints on their possible configurations. It does not require p-value independence: the final probability bound uses the union bound. If there are no true hypotheses, a false rejection is impossible, so the claim also holds in that case.
Assumptions, Diagnostics, and Alternatives
Before using Shaffer’s procedure, check that the family and hypotheses match the derivation. In an all-pairs equality family, the parameter for each arm must refer to the same estimand, and the true equality relations must partition the arms. Each p-value must also be calibrated for its own null hypothesis under the analysis model. For example, a misspecified sampling model can make p-values invalid even if the logical count calculation is correct.
A useful diagnostic is to write down the possible group-size partitions and compute the true-hypothesis counts before looking at the p-values. If the hypotheses do not correspond to pairwise equality of a common parameter, or if the assumed set of counts cannot be justified, do not use the resulting denominators. Also verify that the procedure stops at the first failed comparison. Continuing to test later p-values at their nominal Shaffer thresholds after a failure is not the step-down rule.
Holm’s procedure is a general step-down Bonferroni method for a fixed family of hypotheses and does not need pairwise equality constraints. It is therefore a safer choice when the logical structure is unavailable or uncertain, although it can be more conservative for all-pairs comparisons. Shaffer’s procedure gains power by using attainable true-null counts; it does not use the observed correlation among test statistics. A more tailored closed-testing construction may sometimes exploit additional logical information, but that requires valid local tests for the relevant intersections rather than merely changing the denominators by intuition.
A common mistake is to reason that after one rejection among six pairwise tests, exactly five hypotheses remain and so the next threshold must be \(\alpha/5\). That is Holm’s denominator, not Shaffer’s. For four arms, five true pairwise equalities are impossible: the largest attainable count no greater than five is three. Conversely, it is also a mistake to use the denominator three just because the family contains four arms; the correct denominator depends on the step and the attainable counts. Shaffer’s procedure is a precise count-based rule, not a general license to make thresholds larger.
Check Your Understanding
Use the definitions and step-down rule above. The solutions are included below the questions.
- For four arms, why is a configuration with five true pairwise equality hypotheses impossible?
- For four arms, what denominator does Shaffer’s procedure use at step five, and why?
- For three arms and \(\alpha=0.05\), suppose the sorted p-values are \(0.010,\ 0.030,\ 0.060\). Which hypotheses are rejected?
- State one key condition that would make the all-pairs Shaffer denominators inappropriate.
Solutions. (1) True pairwise equalities form groups. For four arms the possible counts are 6, 3, 2, 1, and 0, so 5 cannot occur. (2) The denominator is 2: two hypotheses remain, and 2 is an attainable count, from a \(2+2\) grouping. (3) The denominators are \(3,1,1\). The first threshold is \(0.05/3=0.01667\), so the first p-value passes; the second threshold is \(0.05\), so \(0.030\) passes; the third threshold is \(0.05\), but \(0.060\) fails. Stop there and reject the first two hypotheses only. (4) For example, the comparisons may not be equality hypotheses for a common arm-specific parameter, so their true relations do not necessarily form the assumed partition of arms.