The Simes Test for an Intersection Null

Why Test the Intersection Null?

In a multi-arm trial, the treatment comparisons form a family: for example, each experimental arm may be compared with a shared control. A global question is whether all the treatment comparisons are null. Testing that question can be more informative than applying a single conservative cutoff to each p-value, while still accounting for the family of tests.

The Simes test answers this global question using the ordered p-values. It is a local test of the intersection null: it gives one decision about whether all component null hypotheses hold. Rejecting that intersection says there is evidence against the claim that every comparison is null; it does not, by itself, establish which treatment comparison is non-null.

Definition: For $m$ treatment comparisons with elementary null hypotheses $H_1,\ldots,H_m$, the intersection null is $H_I=\bigcap_{j=1}^{m}H_j$. Let $p_1,\ldots,p_m$ be valid p-values for those hypotheses, and write their ordered values as $p_{(1)}\le\cdots\le p_{(m)}$. At significance level $\alpha$, the Simes test rejects $H_I$ if at least one ordered p-value satisfies $$ p_{(i)}\le \frac{i\alpha}{m},\qquad i=1,\ldots,m. $$

Deriving the Ordered-P-Value Rule

The test checks a sequence of increasingly permissive thresholds. The smallest p-value is compared with $\alpha/m$, the second-smallest with $2\alpha/m$, and so on, until the largest is compared with $\alpha$. The rule is therefore not “reject if any p-value is below $\alpha$”; it accounts for how many p-values are being examined by making the threshold depend on the rank.

The same rule can be expressed as a single p-value for the intersection test. For each rank $i$, rescale the ordered p-value by $m/i$; the smallest rescaled value is the Simes p-value. It is capped at 1 because a p-value cannot exceed 1.

p_{\text{Simes}}=\min\left\{1,\ \min_{1\le i\le m}\frac{m\,p_{(i)}}{i}\right\}.

Indeed, the rank-$i$ Simes condition is equivalent to $m p_{(i)}/i\le\alpha$. Thus at least one condition holds exactly when $p_{\text{Simes}}\le\alpha$. This equivalence is useful for reporting a single global p-value as well as a reject-or-do-not-reject decision.

Theorem: If, under $H_I$, the component p-values are independent and each is valid (so each true-null p-value is super-uniform), the Simes test has type I error at most $\alpha$. For independent continuous uniform p-values, its rejection probability is exactly $\alpha$. The Simes inequality also gives level-$\alpha$ control under certain positive dependence conditions, commonly expressed as PRDS for the true-null p-values. Marginal p-value validity alone, without an appropriate dependence condition, does not guarantee Simes control.

The calibration statement is the key justification for using these thresholds: under the stated dependence conditions, the probability of crossing any of the rank-specific boundaries when the entire intersection null is true is no greater than $\alpha$. In a multi-arm trial, comparisons sharing a control are generally correlated. The analyst must therefore justify independence or a suitable Simes-valid dependence condition from the joint model or design; it is not enough that every individual p-value is valid on its own.

Worked Decisions in a Multi-Arm Trial

Worked Example: Three Comparisons With a Smallest P-Value Below the First Threshold

Hypothetical data. A trial compares three experimental arms with a control. The prespecified family contains all three two-sided comparisons. The p-values are $0.006$, $0.041$, and $0.120$. Assume the comparison-specific tests are valid under their nulls and the joint dependence satisfies a condition under which the Simes inequality applies. At $\alpha=0.05$, test whether all three comparisons are null.

The ordered p-values are $p_{(1)}=0.006$, $p_{(2)}=0.041$, and $p_{(3)}=0.120$. The three thresholds are $$ \frac{0.05}{3}=0.0167,\qquad \frac{2(0.05)}{3}=0.0333,\qquad \frac{3(0.05)}{3}=0.0500. $$

The first condition is met because $0.006\le0.0167$. The second is not met because $0.041>0.0333$, and the third is not met because $0.120>0.0500$. Since at least one condition holds, reject the intersection null at the 5% level.

The Simes p-value is $$ \min\left\{1,\ 3(0.006),\ \frac{3(0.041)}{2},\ \frac{3(0.120)}{3}\right\} =\min\{1,\ 0.0180,\ 0.0615,\ 0.120\}=0.0180. $$ This is evidence against the claim that all three treatment comparisons are null. It does not, on its own, identify a particular arm as effective.

Worked Example: A Middle Rank Crosses Its Threshold

Hypothetical data. In another three-arm-versus-control trial, the ordered two-sided p-values are $0.020$, $0.030$, and $0.080$. Assume the p-values are valid and the dependence condition for the Simes test is justified. Test the intersection null at $\alpha=0.05$.

The thresholds remain $0.0167$, $0.0333$, and $0.0500$. The first cutoff is not met, since $0.020>0.0167$; the second cutoff is met, since $0.030\le0.0333$; and the third cutoff is not met, since $0.080>0.0500$. Therefore, reject the intersection null.

The Simes p-value is $$ \min\left\{1,\ 3(0.020),\ \frac{3(0.030)}{2},\ \frac{3(0.080)}{3}\right\} =\min\{1,\ 0.0600,\ 0.0450,\ 0.0800\}=0.0450. $$ The global test rejects at the 5% level. The result depends on the second-ranked p-value; checking only whether the smallest p-value is below $0.05/3$ would miss this rejection.

Worked Example: No Ordered P-Value Crosses Its Threshold

Hypothetical data. A trial has four prespecified treatment-versus-control comparisons, with ordered valid p-values $0.014$, $0.031$, $0.060$, and $0.220$. Assume their dependence satisfies the Simes calibration condition. At $\alpha=0.05$, the thresholds are $0.0125$, $0.0250$, $0.0375$, and $0.0500$.

Each comparison of an ordered p-value with its rank-specific threshold fails: $0.014>0.0125$, $0.031>0.0250$, $0.060>0.0375$, and $0.220>0.0500$. Therefore, do not reject the intersection null at the 5% level.

The Simes p-value confirms the decision: $$ \min\left\{1,\ 4(0.014),\ \frac{4(0.031)}{2},\ \frac{4(0.060)}{3},\ \frac{4(0.220)}{4}\right\} =\min\{1,\ 0.0560,\ 0.0620,\ 0.0800,\ 0.220\}=0.0560. $$ Since $0.0560>0.05$, the test does not reject. This is not proof that all treatment effects are zero; it is a failure to find sufficient evidence against the complete intersection null using this test.

Assumptions, Diagnostics, and Interpretation

The Simes test operates on p-values rather than directly on treatment-effect estimates. Its prerequisites therefore include both valid component tests and an acceptable joint dependence structure:

Define the family in advance. The $m$ p-values must correspond to the comparisons included in the intersection null. Adding or dropping comparisons after seeing their results changes the testing problem.
Use valid component p-values. Under each corresponding null, the p-value must have a super-uniform distribution: $P(p_j\le t)\le t$ for $0\le t\le1$. The tests should use the prespecified endpoint, direction, and analysis method.
Check the joint calibration condition. Independence is sufficient, as is a suitable positive-dependence condition for Simes. Shared-control comparisons are not independent merely because their treatment groups are distinct. If the dependence condition is uncertain, the nominal Simes level may not be justified.
Use unrounded p-values for decisions. Rounding can move a value across a cutoff, especially when a p-value is close to $i\alpha/m$. Report enough digits to make the decision reproducible.

A rejection is a family-level conclusion about the all-null intersection. It does not imply that every treatment differs from control, nor that the smallest p-value corresponds to a clinically important effect. The test does not describe effect sizes, uncertainty intervals, or which comparisons deserve follow-up; those require examining the estimates and applying a procedure suited to the intended individual claims.

How Simes Differs From Other Procedures

Bonferroni and Simes both address multiple p-values, but their rules and guarantees differ. Bonferroni rejects the intersection null if any $p_j\le\alpha/m$; its validity follows from the union bound and does not require independence. Simes instead checks the whole ordered sequence, using $i\alpha/m$ at rank $i$. It can reject when no individual p-value meets the first, Bonferroni-sized cutoff, as in the second worked example. That additional sensitivity relies on the Simes dependence condition.

Hochberg’s step-up procedure also uses ordered p-values, but its thresholds are different. With ascending order, its rank-$i$ thresholds are $\alpha/(m-i+1)$, whereas Simes uses $i\alpha/m$. In the second example with $m=3$, Hochberg’s thresholds are $0.0167$, $0.0250$, and $0.0500$; none of $0.020$, $0.030$, or $0.080$ meets its corresponding threshold. Simes rejects there, while Hochberg does not. These are distinct procedures, and any use of a procedure requires its own dependence and error-control conditions.

The Simes test is useful as a compact global test when the scientific question is whether any comparison in a family departs from its null. Its limitations are equally important: its calibration can fail under unsuitable dependence, it is sensitive to the chosen family and p-value construction, and its global conclusion does not identify a particular treatment. A global rejection should not be presented as a set of arm-specific discoveries.

Common Pitfall

Common mistake: “The smallest p-value is greater than $\alpha/m$, so the Simes test cannot reject.” This ignores the higher-rank cutoffs. For example, with three p-values $0.020$, $0.030$, and $0.080$ at $\alpha=0.05$, the first cutoff fails but the second succeeds. Check every rank, or calculate $p_{\text{Simes}}$, before deciding.

Key takeaway: The Simes test rejects the all-null intersection when at least one ordered p-value crosses its rank-specific boundary $i\alpha/m$. Its level-$\alpha$ interpretation depends on valid component p-values and a suitable joint dependence condition.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

For $m=4$ and $\alpha=0.04$, what are the four Simes thresholds?
For $m=3$, the ordered p-values are $0.008$, $0.050$, and $0.090$. Calculate the Simes p-value and state the decision at $\alpha=0.05$.
Why does valid marginal calibration of each p-value not by itself justify the Simes test for comparisons sharing a control?
What does rejection of the intersection null establish, and what does it not establish?

Solutions. (1) The thresholds are $0.04/4=0.0100$, $2(0.04)/4=0.0200$, $3(0.04)/4=0.0300$, and $4(0.04)/4=0.0400$. (2) The rescaled values are $3(0.008)=0.0240$, $3(0.050)/2=0.0750$, and $3(0.090)/3=0.0900$, so $p_{\text{Simes}}=0.0240$. Reject at $0.05$. (3) Dependence affects the probability that one or more ordered p-values cross their thresholds. The Simes level guarantee requires independence or a suitable positive-dependence condition; shared-control comparisons can be correlated, so that condition must be justified. (4) Rejection is evidence against the claim that every comparison is null. It does not show that all comparisons are non-null, identify which comparison is non-null, or establish clinical importance.