Local Tests for Intersection Nulls

Why Local Tests Matter

In a family of clinical trial hypotheses, controlling the chance of even one false rejection requires attention to more than the tests of the individual hypotheses. A test can be well calibrated when a single null hypothesis is considered on its own and still be unsuitable as part of a multiple-testing procedure. The key is to specify a test for each intersection hypothesis and to ensure that every one of those tests meets a common error-rate requirement.

This requirement is especially important when a trial’s analysis includes data-dependent changes. A local test must be valid for the analysis algorithm that will actually be used, including any permitted adaptation—not merely for a simpler fixed design that was considered before adaptation. The relevant guarantee is about the probability of rejection when the intersection null is true.

Definition: Suppose a family contains elementary null hypotheses $H_1,\ldots,H_m$. For each nonempty index set $I\subseteq\{1,\ldots,m\}$, the corresponding intersection hypothesis is $H_I=\bigcap_{i\in I}H_i$. A local test for $H_I$ is a specified statistical test of that intersection. It is a level-$\alpha$ local test if $$ \sup_{\theta\in\Theta_I} P_\theta(\text{reject }H_I)\leq\alpha, $$ where $\Theta_I$ is the set of parameter values under which every hypothesis in $H_I$ is true. The supremum makes the requirement apply to the entire composite null, not just one selected parameter value.

The test for $H_I$ is called “local” because it addresses that particular intersection. Different intersections may use different statistics or rejection rules. The level requirement, however, is the same for all of them: the probability of rejecting any particular intersection when it is true must be at most $\alpha$. A test that is more conservative also qualifies; “level $\alpha$” means no greater than $\alpha$, not necessarily exactly $\alpha$.

For a finite family of $m$ elementary hypotheses there are $2^m-1$ nonempty intersections. Thus, specifying only a test of the intersection of all $m$ hypotheses does not by itself specify local tests for the other intersections. The distinction matters because different configurations of true and false hypotheses make different intersections relevant.

A Standard Construction Using Valid P-Values

A simple way to construct local tests is to use p-values that are valid under their corresponding elementary nulls. A p-value $p_i$ is valid under $H_i$ if, for every parameter value at which $H_i$ is true and every $u\in[0,1]$, $P_\theta(p_i\leq u)\leq u$. This super-uniformity condition permits conservative p-values and does not require independence among tests.

For an intersection $H_I$, a Bonferroni local test rejects when at least one $p_i$, $i\in I$, is no greater than $\alpha/|I|$, where $|I|$ is the number of hypotheses in the intersection. Under $H_I$, all the elementary nulls indexed by $I$ are true. The union bound then gives

P_\theta\left(\min_{i\in I}p_i\leq\frac{\alpha}{|I|}\right) \leq \sum_{i\in I}P_\theta\left(p_i\leq\frac{\alpha}{|I|}\right) \leq \sum_{i\in I}\frac{\alpha}{|I|} =\alpha.

No independence assumption is needed for this argument. The assumptions that do matter are that each p-value is valid under its null and that the test rule and its threshold are specified as stated. In an adaptive trial, p-value validity must hold under the full data-generating and analysis process covered by the design, including the allowed data-dependent changes.

Worked Example: A Bonferroni Test of a Two-Endpoint Intersection

Hypothetical trial. A trial has two elementary null hypotheses, $H_1$ and $H_2$, about two endpoints. Suppose their p-values are valid under their respective nulls and the familywise error target is $\alpha=0.05$. For the intersection $H_{\{1,2\}}=H_1\cap H_2$, the Bonferroni threshold for either p-value is $$ \frac{\alpha}{|I|}=\frac{0.05}{2}=0.025. $$

Suppose the observed p-values are $p_1=0.018$ and $p_2=0.044$. The local test rejects $H_{\{1,2\}}$ if either p-value is at most $0.025$. Since $0.018\leq0.025$, it rejects this intersection.

The level claim does not depend on these observed values. When both nulls are true, the probability of rejection is at most $$ P_\theta(p_1\leq0.025)+P_\theta(p_2\leq0.025) \leq0.025+0.025=0.05. $$ The local test is therefore level $0.05$, provided both p-values are valid under their nulls for the trial’s analysis procedure. The two p-values need not be independent.

The Requirement Across Intersections

A closed testing procedure uses local tests for every nonempty intersection. Its strong familywise error guarantee relies on each such test being level $\alpha$. To see the logic behind this requirement, let $T$ be the set of indices whose elementary null hypotheses are true at a particular parameter value. If at least one elementary null is true, then $T$ is nonempty and $H_T$ is a true intersection. Under the closure rule, rejecting any true elementary hypothesis requires rejecting the intersection containing all true elementary hypotheses, $H_T$. Therefore, the event of at least one false rejection is contained in the event that the local test of $H_T$ rejects. The latter has probability at most $\alpha$ when that local test is level $\alpha$.

This argument explains why the requirement applies to every intersection, not just the intersection of all hypotheses. The true set $T$ can be any nonempty subset: some hypotheses may be true while others are false. A level guarantee only for the all-hypotheses intersection would not provide the needed bound when the true set is smaller.

The condition is a standard sufficient guarantee for closed testing. A particular procedure might, by coincidence, control error even if one local test fails the level condition. But without additional justification, a failed local level check means the general guarantee cannot be invoked.

Worked Example: Checking Weighted Bonferroni Local Tests

Hypothetical trial. Consider three elementary null hypotheses with valid p-values $p_1=0.028$, $p_2=0.022$, and $p_3=0.041$. Use familywise level $\alpha=0.05$ and positive weights $w_1=2$, $w_2=1$, and $w_3=1$. For each intersection $I$, reject if at least one $p_i$, $i\in I$, is at most $\alpha w_i/\sum_{j\in I}w_j$.

For $I=\{1,2\}$, the thresholds are $$ \frac{0.05(2)}{2+1}=0.03333\quad\text{for }p_1, \qquad \frac{0.05(1)}{2+1}=0.01667\quad\text{for }p_2. $$ Since $0.028\leq0.03333$, this local test rejects $H_1\cap H_2$. Its level is at most $0.05$, because under that intersection the union bound gives an upper bound of $0.03333+0.01667=0.05$ on the rejection probability.

For $I=\{2,3\}$, the two thresholds are both $0.025$. The test rejects $H_2\cap H_3$ because $p_2=0.022\leq0.025$; its rejection probability under that intersection is bounded above by $0.025+0.025=0.05$. For $I=\{1,2,3\}$, the thresholds are $0.025$, $0.0125$, and $0.0125$, respectively. None of the observed p-values meets its threshold, so this local test does not reject the all-three intersection.

These decisions illustrate that a p-value is compared with a threshold determined by its intersection, not with one universal threshold across the entire family. The level checks are also intersection-specific: for each $I$, the thresholds sum to $0.05$, so the union bound establishes the required level under $H_I$, assuming the p-values are valid there.

Adaptive Designs: Check the Actual Test

In an adaptive trial, data may influence a prespecified change to the design or analysis. The local-test condition concerns the resulting test as a whole. For every intersection null $H_I$, its rejection probability must be bounded over the null parameter space under the data collection, adaptation, and analysis rules that are actually permitted. A test that has level $\alpha$ under a fixed design does not automatically retain that level after a data-dependent change.

For example, a p-value’s nominal interpretation does not establish its validity if the reported endpoint, test, or stopping rule was selected using the same information in a way the p-value calculation ignores. Validity has to be justified for the selection and adaptation process, not inferred from the p-value’s label or from its behavior in an unadapted plan.

Worked Example: Selection Can Break a Local Test

Hypothetical illustration. Suppose two p-values are independent and uniformly distributed on $[0,1]$ when the global intersection $H_1\cap H_2$ is true. Consider a proposed local test that rejects this intersection if either p-value is at most $0.05$. Its actual rejection probability under the global null is $$ P_\theta(p_1\leq0.05\ \text{or}\ p_2\leq0.05) =1-P_\theta(p_1>0.05,\ p_2>0.05) =1-(0.95)^2=0.0975. $$

The test is not level $0.05$: its rejection probability is $0.0975$. For observed values $p_1=0.030$ and $p_2=0.040$, the proposed test rejects, but a Bonferroni local test compares each p-value with $0.025$ and does not reject. Under the same independence assumption, that Bonferroni test’s rejection probability is $1-(0.975)^2=0.049375$, which is below $0.05$. The Bonferroni level guarantee itself does not require independence; independence was used here only to calculate its exact probability in this illustration.

If the smallest observed p-value is selected and tested at $0.05$, the proposed rule has precisely the same rejection event as “either p-value is at most $0.05$.” The selection cannot be ignored: checking each test’s single-test level does not show that the local test of their intersection is level $0.05$.

Diagnostics, Alternatives, and Limitations

To assess a proposed set of local tests, work intersection by intersection. State the null parameter space, identify the test and its rejection rule, and verify that its rejection probability is at most $\alpha$ throughout that space. If the design is adaptive, include its full adaptation rule in this assessment. A simulation can help diagnose a suspected level violation, but a finite simulation alone is not a proof of a uniform probability bound; its uncertainty and the range of null configurations it covers matter.

Bonferroni local tests are attractive because their validity is simple and does not depend on the dependence structure among p-values. Their cost is that the union bound may be conservative. A test using the joint distribution of the statistics can sometimes be less conservative, but its level guarantee depends on that joint model being correctly justified under the intersection null. Neither approach can substitute a single global test for the tests of all required intersections.

Check the right null: a local test of $H_I$ must be calibrated when all hypotheses in $I$ are true.
Check every intersection: calibration for only the full intersection leaves other possible true configurations unaddressed.
Check the full algorithm: fixed-design calibration alone is not enough when the analysis uses data-dependent changes.
Check the tail probabilities: the validity of each p-value or statistic must be established under the null where it is used.

Common mistake: “Each endpoint test has size $0.05$, so the family of tests has level $0.05$.” This does not follow: the probability of at least one false rejection can exceed $0.05$. The correction is to assess a level-$\alpha$ local test for every intersection, using a justified joint test or a valid multiplicity adjustment such as Bonferroni.

Check Your Understanding

Use the definitions and level criterion above. Each question is followed by a solution so you can check the reasoning.

Question: What does it mean for a local test of $H_I$ to be level $\alpha$?
Solution: Its probability of rejecting $H_I$ must be at most $\alpha$ for every parameter value in $\Theta_I$, or equivalently the supremum of that probability over $\Theta_I$ is at most $\alpha$.
Question: With three valid p-values and $\alpha=0.06$, what Bonferroni threshold is used for each p-value when testing their three-way intersection?
Solution: The threshold is $0.06/3=0.02$. Under the intersection, the union bound limits the rejection probability to at most $3(0.02)=0.06$.
Question: Is independence required for the Bonferroni level argument?
Solution: No. The union bound is valid for dependent events as well as independent ones. Validity of each p-value under its null is required.
Question: Why is a level-$\alpha$ test of only the intersection of all hypotheses insufficient for the general closed-testing guarantee?
Solution: The set of true elementary nulls can be a proper subset of the full family. The relevant true intersection is then the intersection over that subset, so its local test also needs a level-$\alpha$ guarantee.
Question: A test has level $0.05$ under a fixed design. Does this establish level $0.05$ after a data-dependent change to the analysis?
Solution: No. Its rejection probability must be justified under the complete adaptive procedure. The fixed-design result alone does not establish validity after adaptation.