Why the Global Null Is Not Enough

Why the Global Null Is Only One Configuration

In a trial with several treatment arms, the global null says that every treatment comparison is null. It is an important configuration, but it is not the only configuration the trial may encounter. One treatment may have an effect while another does not. If the analysis makes separate claims about each treatment, it must protect against falsely rejecting a null comparison even in such mixed situations.

The key distinction is between controlling error when all null hypotheses are true and controlling error across every possible combination of true and false null hypotheses. The former is called weak familywise error control; the latter is strong familywise error control. The definitions from the earlier material on familywise error are applied here to a concrete three-comparison procedure.

Definition: For hypotheses \(H_1,H_2,H_3\), let \(I_0\) be the set of indices for which the null hypothesis is true in the actual data-generating configuration. If \(R\) is the set of rejected hypotheses, the number of false rejections is \(V=|R\cap I_0|\), and the familywise error rate (FWER) is \(P(V\geq 1)\). Weak control at level \(\alpha\) requires \(P(V\geq 1)\leq\alpha\) when all three null hypotheses are true. Strong control at level \(\alpha\) requires this inequality for every configuration of true and false null hypotheses.

Here, each \(H_i\) can represent a treatment-versus-control null hypothesis. Under the global null, all three treatments satisfy their respective null hypotheses. Under a partial-null configuration, at least one comparison is false and at least one remains true. It is the true nulls in that mixed configuration that can still be falsely rejected.

A Procedure That Passes the Global-Null Check

Consider this deliberately simple rule at nominal level \(\alpha=0.05\): examine the p-value \(p_1\); if \(p_1\leq0.05\), reject all three hypotheses, and otherwise reject none. This is not a recommended analysis. Its purpose is to isolate the logical gap: controlling error when every null is true says nothing, by itself, about what the rule does when one null is false.

For the calculations, assume that under each true null the corresponding p-value is Uniform(0,1), and that the p-values are mutually independent. These assumptions make the probabilities transparent. The counterexample does not rely on dependence complications.

Worked Example: FWER Under the Global Null

Suppose all three treatment comparisons are null. The rule rejects at least one hypothesis exactly when \(p_1\leq0.05\). Because \(p_1\) is Uniform(0,1) under \(H_1\),

\operatorname{FWER} =P(\text{at least one rejection}) =P(p_1\leq0.05) =0.05.

Thus, under the global null, the procedure has FWER exactly 0.05. It passes this particular check, so it has weak control at the 0.05 level. Notice that this calculation does not depend on \(p_2\) or \(p_3\): the rule never uses them to decide whether to reject.

What Changes Under a Partial Null

Now suppose \(H_1\) is false, while \(H_2\) and \(H_3\) are true. A rejection of \(H_1\) is not an error in this configuration, but rejecting either \(H_2\) or \(H_3\) is. Whenever \(p_1\leq0.05\), the rule rejects all three, including both true null hypotheses. If \(H_1\) is often detected, the rule therefore creates many false rejections among the other comparisons.

Worked Example: FWER With Two True Comparisons

Keep \(\alpha=0.05\), and suppose that under the alternative for \(H_1\), the test has power 0.95: \(P(p_1\leq0.05)=0.95\). Let \(H_2\) and \(H_3\) be true, with valid p-values. The rule makes a false rejection if it rejects at least one of these true hypotheses. It does so exactly when \(p_1\leq0.05\), so

\operatorname{FWER} =P(p_1\leq0.05) =0.95.

The procedure’s FWER is 0.95 in this configuration, not 0.05. The numerical value 0.95 is the assumed power for \(H_1\), and it becomes the FWER because each detection of the false \(H_1\) triggers rejection of both true nulls. The true-null p-values can be large; the rule ignores them.

The example uses a particular alternative with 0.95 power to make the calculation concrete. More generally, if the rule rejects all hypotheses when \(p_1\leq0.05\), and \(H_2\) or \(H_3\) is true, then the probability of a false rejection is \(P(p_1\leq0.05)\) under the alternative for \(H_1\). Any power substantially above 0.05 can therefore produce a substantial FWER. This is not a paradox: the global-null calculation concerned a different configuration of truth.

Why the Global-Null Calculation Cannot Settle the Question

The global null is the intersection \(H_1\cap H_2\cap H_3\). A test calibrated to control rejection of that intersection limits the chance of rejecting at least one hypothesis when all three are true. But an individual treatment claim concerns an elementary null such as \(H_2\). When \(H_1\) is false, the global intersection is false even if \(H_2\) is true. Rejecting the global intersection in that situation does not establish that \(H_2\) is false.

The distinction can be expressed directly through the set \(I_0\). Under the global null, \(I_0=\{1,2,3\}\). In the partial-null example, \(I_0=\{2,3\}\). Strong control asks about \(P(R\cap I_0\neq\varnothing)\) for both sets, and for every other possible set of true nulls. Evaluating that probability only for \(I_0=\{1,2,3\}\) leaves the other configurations unexamined.

Key takeaway: A procedure can control FWER under the global null and still have a high probability of rejecting a true treatment comparison when some other comparison is false. A global-null test alone is not a guarantee of strong familywise error control.

Contrasting a Strong-Controlled Rule

A useful contrast is a separate rule that rejects \(H_i\) only when its own p-value is at most a common cutoff \(c\). In the special case where the p-values for true nulls are independent Uniform(0,1), the cutoff can be calibrated so that the probability of one or more rejections under three true nulls is 0.05. This example is an illustration under independence, not a general recommendation for treatment comparisons that may share a control group.

Worked Example: A Common Cutoff Under Independence

With three independent Uniform(0,1) p-values under the global null, the probability of no rejection at cutoff \(c\) is \((1-c)^3\). Set the probability of at least one rejection to 0.05 and solve:

1-(1-c)^3=0.05,\qquad c=1-0.95^{1/3}\approx0.01695.

For a partial-null configuration with two true hypotheses, the probability of rejecting at least one true null is

1-(1-c)^2 =1-(0.95^{1/3})^2 =1-0.95^{2/3} \approx0.03362.

With one true null, the probability is \(c\approx0.01695\); with three true nulls, it is 0.05. Thus, under the stated independence and uniformity assumptions, the procedure controls FWER at 0.05 across all configurations. The calculation works because each true hypothesis is tested using its own p-value and the cutoff was calibrated for the largest number of true nulls. It is not the same as the earlier reject-all rule.

For a numerical illustration, suppose the observed p-values are \(0.004\), \(0.022\), and \(0.047\). At the calibrated cutoff \(c\approx0.01695\), only the first comparison is rejected. If the second and third nulls are true, neither is rejected. In contrast, the reject-all rule would reject all three as soon as it saw \(p_1=0.004\).

The independence condition matters. In a multi-arm trial, treatment-versus-control statistics can be dependent because their comparisons share control participants. The cutoff calculation above is justified only for the stated independent-p-value setup. A valid strong-control procedure for another dependence structure requires justification under that structure; observing that a rule works under the global null is not that justification.

How to Check a Claim of Error Control

When assessing a proposed multiple-comparison rule, do not stop after asking what happens when all treatments are ineffective. Identify the possible sets of true null hypotheses and ask whether the rule protects the true members of each set. In a three-comparison family, the global null is only one of several configurations; mixed configurations are central, not exceptional.

List the comparisons.
State the elementary null hypotheses and identify what each treatment-specific rejection would claim.

Identify the configuration.
Determine which hypotheses are true in the case being assessed. Do not assume that all treatments share the same null status.

Count false rejections.
For each possible outcome, count rejections among true nulls. FWER is the probability that this count is at least one.

Check the assumptions.
Confirm that the null p-values have the stated distributions and dependence. Then evaluate the error probability across configurations, not only under the global null.

Common Pitfalls and Limitations

Equating a global test with treatment-specific evidence. Rejecting the global intersection says that not all null hypotheses hold; it does not identify which individual nulls are false.
Reporting only the all-null error rate. A rule may have FWER at most \(\alpha\) under the global null and much larger FWER under a partial null, as the reject-all example shows.
Confusing a false null with a false rejection. Rejecting a false null is not a familywise error. FWER counts only rejections of hypotheses that are true in the configuration under consideration.
Assuming independence without support. Shared-control comparisons may be dependent. A probability calculation derived under independence cannot automatically be transferred to such a setting.

The central limitation of the global-null check is scope: it assesses one configuration, while strong control is a statement over all configurations. This tutorial does not prescribe a particular adjustment for every MAMS analysis. The appropriate method depends on the hypothesis family, the test statistics, and their dependence. What it does establish is the standard that any claim of strong control must meet.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For three hypotheses, what set of true nulls defines the global null, and what is a partial-null configuration?
A procedure rejects all three hypotheses whenever \(p_1\leq0.05\). Under the global null with a Uniform(0,1) \(p_1\), calculate its FWER.
Under the same rule, suppose \(H_1\) is false with power 0.80 and \(H_2,H_3\) are true. What is the FWER?
Why does a valid test of the global intersection not, by itself, justify rejecting a particular elementary null hypothesis?
Under independent uniform p-values for three true nulls, a common cutoff is \(c=1-0.95^{1/3}\). What is the FWER when exactly one null is true?

Solutions. (1) The global null has \(I_0=\{1,2,3\}\); a partial-null configuration has at least one true and at least one false null, such as \(I_0=\{2,3\}\). (2) The rule rejects at least one hypothesis exactly when \(p_1\leq0.05\), so FWER is \(0.05\). (3) Whenever \(H_1\) is detected, the rule also rejects the true \(H_2\) and \(H_3\); therefore FWER equals the power, \(0.80\). (4) Rejection of the global intersection establishes only that not all nulls hold. A true elementary null can remain true while another comparison is false. (5) With one true null, its p-value is Uniform(0,1), so the FWER is \(c=1-0.95^{1/3}\approx0.01695\), or about 1.695%.