Why Dependence Changes the Šidák Argument
Genomic analyses often test many hypotheses whose test statistics are correlated: nearby variants may be in linkage disequilibrium, and multiple expression measures may share biological or technical sources of variation. The Šidák correction can use a slightly less stringent threshold than Bonferroni, but its usual exact calculation depends on how rejection events overlap. Correlation is therefore not a minor detail: it affects whether the claimed familywise error rate (FWER) is actually controlled.
For a family of \(m\) hypotheses, the FWER is the probability of making at least one rejection among the true null hypotheses. A procedure has strong FWER control at level \(\alpha\) if this probability is at most \(\alpha\) for every configuration of true and false null hypotheses. A \(p\)-value is valid under a true null when \(P(p_i\le u)\le u\) for every \(u\) between 0 and 1.
The Dependence Condition
Under independence, the probability that none of \(m\) true null hypotheses is rejected at threshold \(c\) is \((1-c)^m\), when the null \(p\)-values are continuous uniform. Setting this probability equal to \(1-\alpha\) yields the Šidák threshold. For valid but possibly conservative \(p\)-values, independence gives an upper bound on the FWER of \(1-(1-c)^m\), which is at most \(\alpha\) at \(c=c_{\mathrm{S}}\).
Independence is sufficient, but not necessary. A direct sufficient condition is that, for every subset \(J\) of true null hypotheses and the chosen threshold \(c\), the joint probability of no rejection is at least the product of the marginal probabilities of no rejection:
If each null \(p\)-value is valid, then \(P(p_i>c)\ge 1-c\). Under the stated joint condition, the probability of at least one false rejection is therefore at most \(1-(1-c)^{|J|}\). With \(c=c_{\mathrm{S}}\) and \(|J|\le m\), this is no greater than \(\alpha\). The condition concerns the joint nonrejection events at the threshold used; it should not be replaced by an informal label such as “positively correlated.”
For strong control, the condition must apply to the true-null subset under each configuration being considered. A dependence property established only when every hypothesis is null does not, by itself, establish strong control when some hypotheses are false. In practice, independence of the true-null \(p\)-values across relevant configurations is a clear sufficient assumption. Other dependence structures may also satisfy the joint inequality, but that requires justification for the test statistics and rejection rules actually used.
Worked Example: Independent Null Tests
Suppose a hypothetical analysis tests \(m=100\) valid, independent null hypotheses at familywise level \(\alpha=0.05\). Independence among these null \(p\)-values is assumed, and the calculation is for the case in which all 100 null hypotheses are true.
The Šidák threshold is
Under the assumptions, the probability of at least one rejection is
Thus the familywise error probability is exactly 0.05 for continuous uniform null \(p\)-values, and is no greater than 0.05 for independent valid null \(p\)-values that may be conservative. This conclusion depends on the independence assumption; the numerical threshold alone does not guarantee the result.
A Counterexample Under Arbitrary Dependence
With arbitrary dependence, valid marginal \(p\)-values do not guarantee that the Šidák threshold controls FWER. The following construction gives two individually uniform null \(p\)-values whose rejection events are disjoint. It makes the probability of at least one rejection larger than the target level.
Worked Example: Disjoint Rejection Events Make Šidák Liberal
Consider two true null hypotheses, \(\alpha=0.05\), and a random variable \(U\) uniformly distributed on \([0,1)\). Define \(p_1=U\) and \(p_2=(U+0.5)\bmod 1\), where “mod 1” wraps values at 1 back to 0. Both \(p\)-values are uniform on \([0,1)\), so each is marginally valid. They are dependent.
The Šidák threshold is
The event \(p_1\le c_{\mathrm{S}}\) occurs when \(U\in[0,c_{\mathrm{S}}]\). Since \(c_{\mathrm{S}}<0.5\), the event \(p_2\le c_{\mathrm{S}}\) occurs when \(U\in[0.5,0.5+c_{\mathrm{S}}]\). These intervals are disjoint, and each has probability \(c_{\mathrm{S}}\). Consequently,
So the Šidák threshold fails to control the target FWER in this example, despite both marginal \(p\)-values being valid. The failure comes from the joint distribution of the rejection events, not from a problem with either marginal \(p\)-value.
Bonferroni and Other Dependence Patterns
Bonferroni uses the threshold \(c_{\mathrm{B}}=\alpha/m\). For any dependence structure, the union bound gives
This argument needs valid marginal \(p\)-values but makes no independence assumption. In the two-test counterexample, Bonferroni uses \(c_{\mathrm{B}}=0.05/2=0.025\). The two rejection events remain disjoint, so the FWER is \(2(0.025)=0.05\), exactly the target. In other dependence structures, Bonferroni can be conservative; its guarantee does not require predicting how the tests co-vary.
| Procedure | Threshold for \(m\) tests | Dependence requirement for the stated guarantee |
|---|---|---|
| Šidák | \(1-(1-\alpha)^{1/m}\) | Independence of the relevant null \(p\)-values, or a justified joint nonrejection condition |
| Bonferroni | \(\alpha/m\) | No dependence assumption; each true-null \(p\)-value must be valid |
Worked Example: Perfectly Shared Null Evidence
Suppose four hypothetical tests have identical valid null \(p\)-values: \(p_1=p_2=p_3=p_4=U\), where \(U\) is uniform on \([0,1)\). All four null hypotheses are true, and the target is \(\alpha=0.05\). This construction is an extreme form of dependence.
The Šidák threshold is \(1-(0.95)^{1/4}\approx 0.01274\). Because all four rejection events are identical, the probability of any rejection is
which is below 0.05. Here Šidák is conservative rather than liberal: the tests provide the same null evidence, so four rejection opportunities do not create four separate chances to reject. Bonferroni’s threshold is \(0.05/4=0.01250\), and its FWER is \(0.01250\) in this construction. The example shows why dependence alone does not determine whether Šidák is conservative or liberal. The joint pattern of rejection events matters.
Diagnostics, Practical Choices, and Limitations
In genomic applications, test-statistic correlation matrices, linkage patterns, or shared sample structure can alert an analyst that independence is implausible. They do not, by themselves, establish that the joint nonrejection condition holds. Pairwise correlations are especially incomplete evidence: the event that none of many tests crosses a threshold depends on the joint distribution of the full set, not only on pairs. The dependence induced by one-sided tests can also differ from that induced by two-sided tests on the same statistics.
Before applying Šidák under dependence, specify the null \(p\)-value validity, the subset of hypotheses for which control is claimed, and the joint condition that supports the calculation. If this condition cannot be justified, Bonferroni provides a transparent FWER guarantee under arbitrary dependence. A less conservative dependence-aware procedure may be appropriate when its assumptions and calibration are established, but it should not be inferred merely from observed correlation.
A common mistake is to say, “The tests are positively correlated, so Šidák must be safe.” Correlation is a summary of pairwise linear association, not a general guarantee about joint rejection probabilities. The correction is to verify a dependence result that applies to the actual null \(p\)-values and rejection events; otherwise use a procedure, such as Bonferroni, whose guarantee does not depend on that structure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For \(m=5\) independent valid null \(p\)-values and \(\alpha=0.05\), calculate the Šidák threshold and state the resulting FWER bound.
- Why does the disjoint-event example violate Šidák control even though both \(p\)-values are valid?
- For \(m=8\) tests with arbitrary dependence, what Bonferroni threshold guarantees FWER at most 0.05 when all null \(p\)-values are valid?
- Does observing positive pairwise correlations establish that Šidák is safe? Explain what additional claim is needed.
Solutions. (1) \(c_{\mathrm{S}}=1-(0.95)^{1/5}\approx 0.01021\). Independence gives FWER \(1-(1-c_{\mathrm{S}})^5=0.05\) for continuous uniform null \(p\)-values, and at most 0.05 for independent valid \(p\)-values. (2) Validity constrains each marginal probability \(P(p_i\le c)\), but does not constrain the union enough to give the Šidák bound. Disjoint rejection events make the union probability \(2c_{\mathrm{S}}\approx0.05064\), which exceeds 0.05. (3) The Bonferroni threshold is \(0.05/8=0.00625\); the union bound guarantees FWER at most 0.05 for any dependence. (4) No. Pairwise positive correlations alone do not establish the required joint nonrejection inequality for the relevant true-null tests; justify that condition or choose a dependence-robust method.