True Null Configurations and Error Events

Why the True-Null Configuration Matters

A multiple-testing procedure can produce several rejections, but not every rejection is an error. A rejection is a Type I error only when the corresponding null hypothesis is true. Therefore, to describe familywise error precisely, we need both the procedure's rejection decisions and the set of hypotheses that are true under the parameter configuration being considered.

This distinction matters in adaptive trials. Interim data or other information may affect which hypotheses are tested, which test is applied, or which claims are reported. Such changes can alter the rejection pattern and the probability of an error. They do not, however, make a null hypothesis true or false: truth is determined by the underlying parameter configuration, not by the observed data. The earlier material on strong familywise error control uses this distinction to assess error control across configurations.

Definition: Consider a fixed family of \(m\) elementary null hypotheses \(H_1,\ldots,H_m\). For a parameter value \(\theta\), the true-null set is \(I_0(\theta)=\{i\in\{1,\ldots,m\}:H_i\text{ is true at }\theta\}\). The procedure's rejection set \(R\) contains the indices of the hypotheses it rejects. The number of false rejections is \(V=|R\cap I_0(\theta)|\), and the familywise error event is \(\{V\geq 1\}=\{R\cap I_0(\theta)\neq\varnothing\}\).

The family is the set of claims to which the error guarantee applies. In an adaptive trial, the rejection set \(R\) may depend on the data and on data-dependent choices. For each realized data set, it still identifies the hypotheses rejected by the procedure. To evaluate error control, consider the event that this random rejection set contains at least one index in the true-null set.

The true-null set is specific to a parameter configuration. It may contain every hypothesis, some hypotheses, or none. The procedure's error probability at \(\theta\) is \(P_\theta(R\cap I_0(\theta)\neq\varnothing)\), where the probability is taken over the trial's possible data and any randomization used by the procedure. Strong familywise error control at level \(\alpha\) requires this probability to be at most \(\alpha\) for every parameter configuration under consideration.

Three Kinds of Configurations

The complete-null configuration has \(I_0(\theta)=\{1,\ldots,m\}\): all elementary null hypotheses are true. A partial-null configuration has at least one true and at least one false null, so the true-null set is nonempty but is not the entire family. If \(I_0(\theta)=\varnothing\), there are no true nulls in the family; under the definition above, no rejection can be a Type I error at that configuration.

These cases should not be conflated. Under the complete null, any rejection is an error. Under a partial null, a rejection is an error only if its index belongs to the true-null set. A rejection of a false null is not a Type I error, even if other hypotheses in the family are true.

Worked Example: Classifying a Realized Rejection Pattern

Suppose a hypothetical trial has three elementary hypotheses. At a particular parameter configuration, \(H_1\) and \(H_3\) are true and \(H_2\) is false. Thus \(I_0(\theta)=\{1,3\}\). The procedure rejects \(H_2\) and \(H_3\), so \(R=\{2,3\}\). Determine whether this realized pattern is a familywise error.

First find the false rejections by intersecting the rejection set with the true-null set: \(R\cap I_0(\theta)=\{2,3\}\cap\{1,3\}=\{3\}\). Therefore \(V=|\{3\}|=1\), and \(V\geq1\). This realized pattern is a familywise error: \(H_3\) was rejected even though it is true. Rejecting \(H_2\) is not an error because \(H_2\) is false.

If the same procedure had instead rejected only \(H_2\), then \(R=\{2\}\) and \(R\cap I_0(\theta)=\varnothing\). That pattern would contain no Type I error, despite there being true null hypotheses in the family.

From Individual Rejections to the Error Event

For each index \(i\), let \(A_i=\{i\in R\}\) be the event that \(H_i\) is rejected. For a fixed parameter configuration \(\theta\), familywise error occurs if at least one of the events \(A_i\) occurs for an index in \(I_0(\theta)\). Hence the event can be written as a union:

\{V\geq1\} = \bigcup_{i\in I_0(\theta)} A_i = \{R\cap I_0(\theta)\neq\varnothing\}.

This identity follows directly from the definition of \(V\). The count \(V\) is the number of true-null indices that appear in \(R\). It is at least one exactly when at least one such index is rejected; that is exactly when the intersection is nonempty, or equivalently when one or more of the events \(A_i\) occurs for \(i\in I_0(\theta)\). No independence assumption is needed.

As a consequence, the probability of familywise error is the probability of a union of rejection events over the true nulls. The union bound gives a useful, though not always sharp, inequality:

P_\theta(V\geq1) = P_\theta\!\left(\bigcup_{i\in I_0(\theta)} A_i\right) \leq \sum_{i\in I_0(\theta)}P_\theta(A_i).

This bound does not require the tests to be independent. It also shows why checking only one hypothesis at a time is insufficient: several true-null rejection events can contribute to the chance of at least one error. Conversely, probabilities for hypotheses outside \(I_0(\theta)\) do not enter this Type I error event at that configuration.

Worked Example: A Fixed Rule Under a Partial-Null Configuration

Consider a hypothetical family of three hypotheses. At a particular configuration, \(H_1\) and \(H_3\) are true and \(H_2\) is false. The procedure rejects each hypothesis when its \(p\)-value is at most \(0.025\). Suppose the two true-null \(p\)-values are independent and each is uniformly distributed on \([0,1]\) under this configuration. Find the familywise error probability.

Here \(I_0(\theta)=\{1,3\}\). The familywise error event is that at least one of \(H_1\) or \(H_3\) is rejected. For each true null, the probability of not rejecting is \(1-0.025=0.975\). By the stated independence assumption, the probability that neither true null is rejected is \(0.975\times0.975=0.950625\). Therefore,

P_\theta(V\geq1) = 1-P_\theta(p_1>0.025,\ p_3>0.025) = 1-(0.975)^2 = 0.049375.

The familywise error probability is \(0.04938\), or about \(4.94\%\). The \(p\)-value for the false null \(H_2\) is irrelevant to this calculation: rejecting \(H_2\) does not create a Type I error at this configuration. Independence was used to multiply the two nonrejection probabilities; without it, the event definition would remain valid, but that multiplication would not be justified.

When the Rejection Rule Depends on Data

In an adaptive procedure, \(R\) is a data-dependent set. For example, the procedure may select one endpoint from several candidates and then reject only the selected endpoint if its test crosses a threshold. The same event definition still applies: familywise error occurs if any rejected index belongs to \(I_0(\theta)\). What changes is the probability of that event, because the selection rule changes which rejection events can occur and how they are related.

In particular, a test's nominal error rate when considered alone does not automatically give the error rate after it has been selected using the same data. Selection can make the chosen test disproportionately likely to have an unusually small \(p\)-value. The following example isolates that issue under explicit assumptions.

Worked Example: Selecting the Smaller of Two True-Null P-Values

Suppose two candidate hypotheses are both true. Under the hypothetical model, their \(p\)-values \(p_1\) and \(p_2\) are independent and each is uniformly distributed on \([0,1]\). An adaptive rule selects the hypothesis with the smaller \(p\)-value and rejects that selected hypothesis if its \(p\)-value is at most \(0.05\). Calculate the familywise error probability.

Because both null hypotheses are true, either possible rejection is an error. The procedure makes an error exactly when the smaller \(p\)-value is at most \(0.05\), which is the event \(\min(p_1,p_2)\leq0.05\). Its complement is that both \(p\)-values exceed \(0.05\). Independence gives

P(V\geq1) = 1-P(p_1>0.05,\ p_2>0.05) = 1-(0.95)^2 = 0.0975.

The familywise error probability is \(0.0975\), or \(9.75\%\), not \(5\%\). Each \(p\)-value considered on its own is uniform under its true null, but the selection rule chooses the smaller one. The calculation depends on both hypotheses being true and on the stated independence assumption; the event definition itself requires neither assumption.

How to Check a Configuration and Its Error Event

A clear analysis separates the fixed inferential family, the parameter configuration, and the procedure's decisions. A practical way to do that is to record the true-null set and the rejection set separately before deciding whether an error occurred.

Specify the family.
List the elementary hypotheses that count as possible claims. State how data-dependent selections or adaptations map to decisions about these hypotheses.

Specify the configuration.
For a parameter value \(\theta\), identify \(I_0(\theta)\), the indices of all true elementary nulls. Do not infer truth from the observed \(p\)-values.

Record the rejection set.
For a realized data set, list \(R\), including any rejected claim produced after an adaptive selection.

Check the intersection.
Compute \(R\cap I_0(\theta)\). The realized outcome is a familywise error if and only if this set is nonempty.

For theoretical evaluation, \(\theta\) is varied across configurations and probabilities are computed under each one. In an actual trial, the true configuration is generally not known; error control is a property of the procedure's operating characteristics, not a label that can usually be attached with certainty to one observed result. Diagnostics should therefore focus on whether the family and rejection rule have been described completely and whether calculations include every true-null rejection event permitted by the rule.

Common Mistakes and Limitations

Counting every rejection as an error. That is correct only under the complete-null configuration. Under a partial-null configuration, intersect \(R\) with \(I_0(\theta)\) to identify the false rejections.
Checking only the overall null. A procedure's behavior when all nulls are true does not by itself characterize its behavior under partial-null configurations. Strong control concerns every configuration in the scope of the guarantee.
Treating an untested hypothesis as automatically rejected. A hypothesis not rejected is not in \(R\). But if an adaptive rule makes a claim for a selected hypothesis, that claim must be represented in the rejection set.
Using a nominal single-test level after selection. The probability of rejection after data-dependent selection need not equal the probability for a test considered in isolation. Derive the error event from the actual selection and rejection rule.
Assuming independence without justification. The set-based event and its union representation hold regardless of dependence. Multiplying probabilities of component events requires an appropriate independence assumption.

The set representation does not, by itself, provide a testing procedure or establish a numerical error bound. To assess a particular adaptive design, one must also know the rule that generates \(R\) and the probability distribution of the data under each configuration. Its value is that it makes the target error event unambiguous before those probabilities are calculated.

Key takeaway: At a fixed parameter configuration, familywise error occurs exactly when the rejection set contains at least one true-null index: \(R\cap I_0(\theta)\neq\varnothing\). Adaptive rules can change the probability of this event, but not its definition.

Check Your Understanding

Answer each question by identifying the true-null set and the rejection set where relevant.

A family has four hypotheses, with \(I_0(\theta)=\{1,4\}\). The procedure rejects \(H_2\) and \(H_3\). Is this a familywise error?
In the same family and configuration, the procedure rejects \(H_2\) and \(H_4\). What is \(R\cap I_0(\theta)\), and is there a familywise error?
What is the true-null set under the complete-null configuration? What does the familywise error event reduce to?
Why does the event identity for familywise error not require independence among tests?
Two independent true-null \(p\)-values are each uniform on \([0,1]\). A rule rejects if either is at most \(0.02\). Calculate the familywise error probability.

Solutions: (1) No. Here \(R=\{2,3\}\), so \(R\cap I_0(\theta)=\{2,3\}\cap\{1,4\}=\varnothing\). (2) \(R=\{2,4\}\), so \(R\cap I_0(\theta)=\{4\}\); this is a familywise error because at least one true null is rejected. (3) The true-null set is the entire family, \(\{1,2,3,4\}\); any rejection is then a familywise error. (4) The identity follows from the definition of \(V\): \(V\geq1\) exactly when at least one true-null hypothesis is rejected. This logical equivalence does not depend on the joint distribution of the tests. (5) The error event is that at least one \(p\)-value is at most \(0.02\). By independence, its probability is \(1-(1-0.02)^2=1-(0.98)^2=0.0396\), or \(3.96\%\).