Tutorials › Statistical Theory III - Theories of Inference, Multivariate Methods, and Contingency Tables › Power and Error Criteria Under Familywise Control

Home  /  Tutorials  /  Simultaneous Inference and Power

Simultaneous Inference and Power · Tutorial 284 of 972

Power and Error Criteria Under Familywise Control

Distinguish the chance of detecting individual contrasts, at least one false null, or every false null from the probability of making any false rejection.

Advanced 10 min read

What You'll Learn

  • Define marginal power for an individual contrast test at a specified parameter value.
  • Define disjunctive and conjunctive power using the set of false null hypotheses.
  • Relate familywise error control to the probability of any rejection under the global null.
  • Calculate joint power criteria for independent tests and identify when independence matters.
  • Use general bounds to connect marginal probabilities to disjunctive and conjunctive power.
  • Avoid confusing a correct detection with a rejection of any hypothesis in the family.

Why a Family Has More Than One Power

A family of contrast tests can be successful in several different senses. An investigator may want a high chance of detecting one particular contrast, a high chance of detecting at least one of several nonzero contrasts, or a high chance of detecting every nonzero contrast. These are different objectives, even when the same tests and rejection thresholds are used.

They must also be separated from familywise error control. Power concerns rejections of false null hypotheses; familywise error concerns rejections of true null hypotheses. Under the global null, every hypothesis is true, so the probability of any rejection is an error probability—not power.

Definition: Consider a family of \(m\) contrast hypotheses \(H_{01},\ldots,H_{0m}\), with rejection events \(R_1,\ldots,R_m\). At a parameter value \(\theta\), let \(I_1(\theta)\) be the indices of false null hypotheses and \(I_0(\theta)\) the indices of true null hypotheses. For \(j\in I_1(\theta)\), the marginal power of test \(j\) is \(\pi_j(\theta)=P_\theta(R_j)\). The disjunctive power is \(P_\theta(\bigcup_{j\in I_1(\theta)}R_j)\), the probability of rejecting at least one false null. The conjunctive power is \(P_\theta(\bigcap_{j\in I_1(\theta)}R_j)\), the probability of rejecting every false null. These are defined when \(I_1(\theta)\) is nonempty.

The false-null set matters. If only one contrast is nonzero, disjunctive and conjunctive power both reduce to that contrast's marginal power. If several nulls are false, the criteria generally differ. Rejection of a true null is not a successful detection of a false null, even if it occurs alongside a correct rejection.

Definition: The familywise error rate at \(\theta\) is \(P_\theta(\bigcup_{j\in I_0(\theta)}R_j)\), the probability of rejecting at least one true null. Strong familywise error control at level \(\alpha\) requires this probability to be at most \(\alpha\) for every parameter configuration. Under the global null, \(I_0(\theta)=\{1,\ldots,m\}\), so the familywise error rate is \(P_\theta(\bigcup_{j=1}^m R_j)\).

The final statement follows directly from the definition: under the global null, every possible rejection is false. Control only under the global null is often called weak familywise control; strong control covers configurations with both true and false nulls. The distinction is important because a procedure's power under alternatives is not guaranteed by its error control under the global null.

Worked Example: Two Independent Contrast Tests

Worked Example: Two Independent Contrast Tests

Suppose two one-sided tests concern contrasts \(\theta_1\) and \(\theta_2\), with \(H_{0j}:\theta_j\leq 0\) against \(H_{1j}:\theta_j>0\). Each test rejects when its standardized statistic exceeds \(1.645\). Assume the statistics are independent and, at the parameter value considered, \(Z_1\sim N(1,1)\) and \(Z_2\sim N(2,1)\). Both null hypotheses are false.

For a statistic \(Z_j\sim N(\delta_j,1)\), the rejection probability is \(P(Z_j>1.645)=1-\Phi(1.645-\delta_j)\), where \(\Phi\) is the standard normal distribution function. Thus the marginal powers are

$$ \pi_1=1-\Phi(0.645)\approx 0.2595,\qquad \pi_2=1-\Phi(-0.355)\approx 0.6387. $$

Disjunctive power is the chance that at least one test rejects. Independence lets us calculate its complement as the probability that neither rejects:

$$ P(\text{neither rejects}) =\Phi(0.645)\Phi(-0.355) \approx (0.7405)(0.3613) \approx 0.2676. $$

Therefore the disjunctive power is \(1-0.2676\approx 0.7324\), using the unrounded product \(0.2675519\) before rounding the result. Conjunctive power requires both tests to reject. By independence,

$$ P(R_1\cap R_2)=\pi_1\pi_2 \approx (0.2595)(0.6387) \approx 0.1657. $$

The probability of detecting at least one contrast is much larger than the probability of detecting both. Neither number is a familywise error rate here: both null hypotheses are false, so there are no true nulls whose rejection could count as a familywise error.

How the Criteria Relate

Write \(\pi_j=P_\theta(R_j)\) for each false null at the parameter value under discussion, and let \(k\) be the number of false nulls. The joint criteria cannot usually be recovered from the marginal powers alone, because the rejection events may be dependent. There are nevertheless useful bounds that hold without independence.

Theorem: For \(k\geq 1\) false null hypotheses, $$ \max_{j\in I_1(\theta)}\pi_j \leq P_\theta\!\left(\bigcup_{j\in I_1(\theta)}R_j\right) \leq \min\!\left(1,\sum_{j\in I_1(\theta)}\pi_j\right), $$ and $$ \max\!\left(0,\sum_{j\in I_1(\theta)}\pi_j-(k-1)\right) \leq P_\theta\!\left(\bigcap_{j\in I_1(\theta)}R_j\right) \leq \min_{j\in I_1(\theta)}\pi_j. $$

Proof. For each false-null index \(j\), \(R_j\) is a subset of the union of the false-null rejection events. Taking probabilities gives \(\pi_j\leq P_\theta(\bigcup R_j)\), and hence the lower bound by taking the largest \(\pi_j\). The union bound gives \(P_\theta(\bigcup R_j)\leq\sum_j\pi_j\); a probability is also at most one, proving the upper bound.

The intersection is a subset of each event \(R_j\), so its probability is at most every \(\pi_j\), and therefore at most their minimum. For the lower bound, the complement of the intersection is the union of the complements. Applying the union bound to those complements gives

$$ P_\theta\!\left(\bigcap_jR_j\right) =1-P_\theta\!\left(\bigcup_jR_j^c\right) \geq 1-\sum_jP_\theta(R_j^c) =\sum_j\pi_j-(k-1). $$

Since probabilities are nonnegative, the lower bound can be replaced by its maximum with zero. This proves both bounds. \(\square\)

If the false-null rejection events are independent, the bounds sharpen to exact formulas:

$$ P_\theta\!\left(\bigcup_{j\in I_1(\theta)}R_j\right) =1-\prod_{j\in I_1(\theta)}(1-\pi_j), \qquad P_\theta\!\left(\bigcap_{j\in I_1(\theta)}R_j\right) =\prod_{j\in I_1(\theta)}\pi_j. $$

The first identity follows by taking the complement of the union: none of the tests rejects, an event with probability \(\prod_j(1-\pi_j)\) under independence. The second is the product rule for independent rejection events. Without independence, multiplying marginal probabilities is not justified.

Worked Example: Three Tests Under the Global Null

Suppose three independent one-sided tests are each conducted at level \(0.05\), and all three null hypotheses are true. Each test has rejection probability \(0.05\). The probability of no rejections is \((1-0.05)^3=0.95^3=0.857375\). Thus

$$ \text{FWER}=P(\text{at least one rejection}) =1-0.95^3=0.142625. $$

The individual tests each have the stated \(0.05\) size, but the familywise error rate is about \(0.1426\), not \(0.05\). There are no false nulls in this example, so disjunctive and conjunctive power are not defined as probabilities of detecting false nulls; every rejection is an error.

For comparison, allocate \(\alpha/3=0.05/3=1/60\) to each test. Under independence, the probability of at least one rejection is then

$$ 1-\left(1-\frac{1}{60}\right)^3 =1-\left(\frac{59}{60}\right)^3 =\frac{10621}{216000} \approx 0.04917. $$

This is below \(0.05\). More generally, the union bound ensures familywise error at most \(0.05\) when each of three tests has size at most \(0.05/3\), without requiring independence. The example concerns error control, not power: under the global null there is no false null to detect.

Worked Example: One True Null and Two False Nulls

Worked Example: One True Null and Two False Nulls

Suppose a family has three tests. At a particular parameter value, \(H_{01}\) is true and \(H_{02},H_{03}\) are false. Assume the tests' rejection events are independent, with probabilities \(0.05\), \(0.80\), and \(0.60\), respectively. The \(0.05\) is the rejection probability for the true null; the other two values are marginal powers.

Disjunctive power counts a rejection of either false null, so

$$ 1-(1-0.80)(1-0.60)=1-(0.20)(0.40)=0.92. $$

Conjunctive power requires both false nulls to be rejected:

$$ (0.80)(0.60)=0.48. $$

Familywise error is a different probability. Only the first null is true, so the familywise error rate at this parameter value is \(P(R_1)=0.05\). The probability of any rejection in the family is instead

$$ 1-(1-0.05)(1-0.80)(1-0.60) =1-(0.95)(0.20)(0.40) =0.924. $$

That \(0.924\) includes correct detections of false nulls and a possible false rejection of the true null. In fact, the probability of rejecting \(H_{01}\) while missing both false nulls is \((0.05)(0.20)(0.40)=0.004\). Thus the probability of any rejection is not the disjunctive power, even though both refer to rejection events in the same family.

Choosing the Criterion and Avoiding a Common Mistake

The appropriate power criterion follows from the scientific objective. If one contrast is primary, its marginal power is directly relevant. If the goal is to find at least one effect among several, disjunctive power matches that goal. If a study is considered successful only when every targeted effect is detected, conjunctive power is the relevant quantity. For a family containing both true and false nulls, report the false-null set or state the parameter configuration; otherwise, the meaning of a joint power number is ambiguous.

A common mistake is to say, “The probability of rejecting at least one hypothesis is the power of the procedure.” That reasoning treats every rejection as a successful discovery. Under the global null, however, every rejection is false, and the probability of any rejection is precisely the familywise error rate. Under an alternative with both true and false nulls, any rejection may include a false rejection, while disjunctive power counts only rejections among the false nulls.

Another mistake is to calculate conjunctive power by multiplying marginal powers without checking dependence. The product is correct only when the relevant rejection events are independent. For correlated contrast statistics, joint probabilities must account for their dependence; the marginal values alone generally do not determine them.

Key takeaway: Marginal power concerns one false null, disjunctive power concerns detecting at least one false null, and conjunctive power concerns detecting all false nulls. Familywise error control instead limits the probability of rejecting any true null, with the global-null case making every rejection an error.

Check Your Understanding

Use the false-null set explicitly when identifying a power criterion.

  1. At a parameter value where all four null hypotheses are false, what event defines disjunctive power?
  2. Under the global null, explain why the probability of any rejection is familywise error rather than power.
  3. Two false-null tests have independent marginal powers \(0.70\) and \(0.50\). Calculate their disjunctive and conjunctive power.
  4. If only one null in a family is false, how do disjunctive and conjunctive power relate to its marginal power?
  5. Why is the product of marginal powers generally not a valid calculation of conjunctive power for correlated tests?

Previous

Next

Back to Tutorials
Clinical Biostats