Šidák Correction Under Independence

Why Independence Changes the Threshold

In a genomic analysis, a testing family may contain many hypotheses: for example, one null hypothesis for each variant tested for association with a phenotype. If each test is conducted at level \(\alpha\), the probability of at least one false rejection across the family can be much larger than \(\alpha\). Familywise error control addresses this probability. The Šidák correction uses a specific feature of the testing situation—independence—to obtain a per-test threshold from the probability that none of the tests produces a false rejection.

This is different from setting a threshold by adding upper bounds on individual error probabilities. Under independence, the probability of no false rejection factors into a product. That product gives an exact familywise threshold when the null p-values are uniform, and a conservative threshold when they are merely valid. The distinction matters: Šidák is not a general replacement for methods that control familywise error without an independence assumption.

Deriving the Šidák Cutoff

Suppose first that all \(m\) null hypotheses are true, and that their p-values are mutually independent and uniformly distributed on \([0,1]\). For a common per-test cutoff \(c\), the probability that test \(i\) does not reject is \(P(p_i>c)=1-c\). Independence means the probability that no test rejects is the product of these probabilities. Therefore the probability of at least one rejection is one minus that product.

To make this familywise error equal to the target \(\alpha\), solve \(1-(1-c)^m=\alpha\) for \(c\). This gives the Šidák threshold. Under the assumptions above, the FWER is exactly \(\alpha\) when all \(m\) null hypotheses are true.

Exact uniformity is not essential for control. A p-value is valid under its null if \(P(p_i\leq t)\leq t\) for every \(t\) between 0 and 1. For example, discrete tests can yield null p-values that are conservative rather than exactly uniform. If the p-values for the true null hypotheses are independent and valid, then \(P(p_i>c)\geq 1-c\), so the probability of no false rejection among \(m_0\) true nulls is at least \((1-c)^{m_0}\). With the Šidák threshold, this implies FWER no greater than \(\alpha\), even when some hypotheses are false.

where \(I_0\) is the set of true null hypotheses and \(m_0\) is its size. The first inequality uses independence and null validity; the second uses \(m_0\leq m\). Thus, mutual independence of the tests (and hence of their p-values) together with valid p-values for true nulls gives strong FWER control: control under any configuration of true and false null hypotheses. When all p-values are exactly uniform under their nulls, the bound for the true-null subset is exact at \(1-(1-c)^{m_0}\).

Worked Šidák Calculations

Adjusted P-Values and the Bonferroni Comparison

For a single-step Šidák procedure applied to \(m\) tests, the adjusted p-value for an observed \(p_i\) is

The cap at 1 is needed because an adjusted p-value cannot exceed 1. The formula comes from solving \(1-(1-p_i)^m=\alpha\) for the familywise level at which the observed p-value would just meet the Šidák cutoff. Consequently, \(p_i\leq c_{\text{Sidak}}\) is equivalent to \(p_{i,\text{adj}}\leq\alpha\).

For \(0<\alpha<1\) and \(m>1\), the Šidák cutoff is at least as large as the Bonferroni cutoff \(\alpha/m\), and is strictly larger except at boundary cases. One way to see this is that \(x^{1/m}\leq 1+(x-1)/m\) for \(0<x\leq1\), so \(1-(1-\alpha)^{1/m}\geq\alpha/m\). The difference is often small when \(\alpha\) is small, but it can matter when p-values lie close to the thresholds.

Bonferroni and Holm control FWER without requiring independence, though they can be conservative. Šidák uses independence to obtain its product calculation. The larger Šidák cutoff is not a general improvement: if the tests are dependent, the product calculation may fail and the intended error control may not hold. The later material on dependence considers that setting; for this procedure, the key decision is whether independence is defensible.

Assumptions, Diagnostics, and Common Pitfalls

The calculation requires both a correctly specified testing family and valid individual tests. Independence concerns the joint behavior of tests, not whether their hypotheses happen to be different. In genomic data, tests at nearby variants may be dependent because of linkage disequilibrium. Dependence can also arise when tests use overlapping samples, shared outcomes, or common estimated quantities. A lack of obvious correlation in a small observed set is not proof of independence; and zero correlation alone does not generally establish independence.

Before using Šidák, check that the family was specified for the scientific question, that each null p-value is valid under its test assumptions, and that the test outcomes can reasonably be treated as mutually independent. In a design with known independent groups or independent data streams, independence may be plausible. With correlated genomic measurements or shared participants, it needs careful justification rather than assumption by convenience. If independence is doubtful, do not describe the Šidák calculation as exact; use an approach whose guarantees match the dependence structure.

• Using \(\alpha/m\) as the Šidák cutoff. That is the Bonferroni cutoff. The Šidák cutoff is \(1-(1-\alpha)^{1/m}\).
• Assuming independence because the tests have different labels. Distinct hypotheses can still have dependent test statistics and p-values.
• Claiming exact control for conservative p-values. Valid p-values and independence guarantee control at or below \(\alpha\); exact equality requires uniform null p-values and, for the full-family expression, all nulls true.
• Changing the family after seeing the results. The value of \(m\) must correspond to the testing family for which familywise control is being claimed.

A practical diagnostic is to trace how each test is constructed: which observations, phenotypes, covariates, and genomic measurements it uses. Shared inputs are a warning that independence may be implausible. Simulation under a defensible null model can help reveal joint behavior, but a limited simulation cannot by itself establish independence.