Why Adjust for a Family of Tests?
When several hypotheses are tested, each at significance level \(\alpha\), the chance of making at least one Type I error across the collection can exceed \(\alpha\). The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in a specified family. The Šidák adjustment chooses a per-test significance level to control that probability, using independence among the tests.
The key point is that the Šidák formula is not a universal correction for any collection of tests. Its exact equality relies on independent, exactly uniform p-values under their null hypotheses. If null p-values are independent but only valid, the same calculation gives an upper bound, not necessarily an equality. If the tests are dependent, the independence-based calculation need not control FWER; Bonferroni control does not require independence.
Deriving the Šidák Adjustment
Suppose a family contains \(m\) tests, and each test will be rejected when its p-value is at most \(c\). Begin with the complete-null case: all \(m\) null hypotheses are true. If the p-values are independent and exactly uniform under their nulls, each test has probability \(c\) of rejection. The probability that none is rejected is the product of the individual non-rejection probabilities, \((1-c)^m\). Therefore the probability of at least one rejection is \(1-(1-c)^m\).
Choose a desired upper limit \(\alpha\) for the probability of at least one false rejection among the \(m\) tests.
For independent tests with exact uniform null p-values, the probability of no false rejection at per-test level \(c\) is \((1-c)^m\).
Solve \(1-(1-c)^m=\alpha\) for \(c\), giving \(c=1-(1-\alpha)^{1/m}\).
Thus, the single-step Šidák procedure rejects test \(j\) when \(p_j\le\alpha_{\text{Sidak}}\). Under independent, exactly uniform null p-values, its FWER under the complete null is exactly \(\alpha\). The method is also used to control FWER when only some null hypotheses are true, provided the p-values corresponding to the true nulls are independent and valid. The independence and validity conditions matter in that setting, as the bound below shows.
To justify the first inequality, let \(p_1,\ldots,p_k\) denote the p-values for the true nulls. Validity gives \(P(p_j\le c)\le c\), and hence \(P(p_j>c)\ge 1-c\). Independence gives \(P(p_1>c,\ldots,p_k>c)=\prod_{j=1}^kP(p_j>c)\ge(1-c)^k\). Taking the complement proves the claimed upper bound. Since \(k\le m\), \(1-(1-c)^k\le1-(1-c)^m\). This establishes control even if some hypotheses in the family are false, as long as the true-null p-values meet the stated conditions.
Worked Example: Setting a Per-Test Level for Four Tests
Worked Example: Setting a Per-Test Level for Four Tests
A researcher plans four tests of separate outcomes and wants a familywise error target of \(\alpha=0.05\). Assume that under the complete null the four p-values are independent and exactly uniform. Find the Šidák cutoff and verify the familywise error rate. Compare the cutoff with the Bonferroni cutoff.
Method. Apply the Šidák formula with \(m=4\) and \(\alpha=0.05\):
Each test is therefore rejected when its p-value is at most approximately \(0.0127\). The familywise probability under the stated exact-independence model is
The Bonferroni cutoff is \(\alpha/m=0.05/4=0.0125\), slightly smaller. Both cutoffs control FWER under their respective assumptions; Bonferroni remains valid without independence. The calculation avoids the common mistake of using \(0.05\) for each of the four tests and treating the familywise error as though it were still \(0.05\).
Adjusted p-Values and the Validity Qualification
For a single-step Šidák procedure, a convenient family-adjusted p-value for an observed individual p-value \(p\) is \(1-(1-p)^m\), capped at 1 if necessary. Comparing this adjusted value with \(\alpha\) is equivalent to comparing \(p\) with \(1-(1-\alpha)^{1/m}\): both reject precisely when at least one individual p-value is no larger than the Šidák cutoff. Under independent, exactly uniform null p-values, the transformation gives the exact complete-null probability of seeing at least one p-value no larger than \(p\). With independent valid p-values, it instead gives a conservative upper bound.
Worked Example: Adjusting a Smallest P-Value
Five independent tests are conducted, and the smallest observed p-value is \(0.010\). Assume first that the five null p-values are independent and exactly uniform. Calculate the Šidák-adjusted p-value for the smallest result and decide whether it crosses a familywise target of \(0.10\). Then state what changes if the p-values are independent but only valid.
Method. For the smallest p-value, \(p=0.010\), and \(m=5\):
Compute the power step by step: \(0.99^2=0.9801\), \(0.99^3=0.970299\), \(0.99^4=0.96059601\), and \(0.99^5=0.9509900499\). Thus \(p_{\text{adjusted}}=0.0490099501\), or \(0.0490\) to four decimal places. It is below \(0.10\), so the result crosses the familywise target under the specified exact-independence model.
If the p-values are independent but only valid, then each true-null probability of a p-value no larger than \(0.010\) is at most \(0.010\), not necessarily exactly \(0.010\). The probability of at least one such true-null p-value among all five is therefore at most \(1-0.99^5=0.0490099501\). The adjusted value is a conservative bound in that case; calling it the exact probability would be unjustified. The common mistake is to replace the validity inequality \(P(p\le t)\le t\) with equality without establishing exact uniformity.
When Independence Is Not Available
The Šidák derivation uses a product of non-rejection probabilities. Knowing only the separate p-value distributions is not enough to factor the joint probability in this way. Tests can be dependent because they use overlapping observations, correlated outcomes, or a shared estimated quantity. In ANOVA contrast work, for example, the independence of the actual test results must be justified under the model and the test construction; orthogonal contrast coefficients alone should not be used as a blanket substitute for checking the p-value dependence structure.
Bonferroni control provides a dependence-robust alternative. If each of \(m\) tests uses cutoff \(\alpha/m\), then the probability of at least one false rejection is at most the sum of the individual true-null rejection probabilities, which is at most \(m(\alpha/m)=\alpha\). This union-bound argument does not require independence. Under independence, the Šidák cutoff is slightly larger than \(\alpha/m\) for \(m>1\), so it can be less conservative; when independence is uncertain, that advantage does not justify assuming it.
| Procedure | Per-test cutoff | Condition for the stated control |
|---|---|---|
| Šidák | \(1-(1-\alpha)^{1/m}\) | Independence of the relevant true-null p-values, together with validity; equality requires exact uniformity under the complete null |
| Bonferroni | \(\alpha/m\) | Valid individual p-values; no independence assumption |
Worked Example: Why Dependence Changes the Calculation
Consider two true null hypotheses with p-values \(p_1=U\) and \(p_2=1-U\), where \(U\) is uniform on \([0,1]\). Each p-value is exactly uniform by itself, but the pair is dependent. For a familywise target of \(0.05\), calculate the FWER from the Šidák cutoff and compare it with Bonferroni.
Method. The Šidák cutoff for \(m=2\) is
A false rejection occurs if \(U\le c\) or \(1-U\le c\), which is equivalent to \(U\le c\) or \(U\ge1-c\). These two events are disjoint because \(c<0.5\), and each has probability \(c\). Therefore the actual FWER is
This exceeds \(0.05\), so the independence-based Šidák cutoff does not control FWER in this example. Bonferroni uses \(0.05/2=0.025\); the two rejection regions are again disjoint, giving FWER \(0.025+0.025=0.050\). The error to avoid is treating individually uniform p-values as though their joint distribution must be independent.
Practical Checks Before Applying the Adjustment
Before using a Šidák calculation, identify the family of hypotheses whose combined error is to be controlled, count the tests in that family, and specify the familywise target. Then check whether independence applies to the p-values for the true null hypotheses under consideration. The distribution of the test statistics, the data structure, and any shared estimated variance can all matter. If independence has not been established, Bonferroni is the straightforward dependence-robust choice.
Keep three claims separate: exact independent uniform null p-values give the Šidák equality; independent valid null p-values give the Šidák upper bound; and arbitrary dependence does not justify either independence calculation. This distinction is central to using the adjustment correctly and is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each question, state any relevant independence or validity condition before interpreting the adjustment.
- Four independent tests have exact uniform p-values under their nulls. Find the Šidák per-test cutoff for familywise target \(0.05\), to four decimal places. (Answer: \(1-0.95^{1/4}\approx0.0127\).)
- Eight tests are to be controlled at familywise target \(0.01\). Find the Šidák per-test cutoff, to six decimal places. (Answer: \(1-0.99^{1/8}\approx0.001256\).)
- For four independent tests, the smallest observed p-value is \(0.020\). Find its single-step Šidák-adjusted value, to four decimal places. (Answer: \(1-0.98^4=0.07763184\approx0.0776\).)
- Two independent true-null p-values are valid but not known to be exact. Both are tested at \(0.030\). Give the Šidák upper bound for the probability of at least one false rejection, to four decimal places, and say whether it must be an equality. (Answer: At most \(1-0.97^2=0.0591\); equality is not guaranteed.)
- Two p-values are \(U\) and \(1-U\), with \(U\) uniform on \([0,1]\), and both nulls are true. At target \(0.05\), calculate the actual FWER using the two-test Šidák cutoff, to four decimal places. (Answer: \(2(1-\sqrt{0.95})\approx0.0506\), so the target is exceeded.)