Šidák Thresholds Under Independent Tests

Why Independence Changes the Threshold

When a genome-wide association analysis tests many variants, a small per-test significance level does not by itself describe the chance of making at least one false rejection across the testing family. The family-wise error rate (FWER) is the probability of at least one false rejection among the hypotheses in that family. Bonferroni control bounds this probability without requiring independence. If the tests are independent, however, the probability of avoiding a false rejection can be calculated directly. This gives the Šidák threshold.

The distinction is useful but conditional: the Šidák calculation relies on independence of the relevant tests under their null hypotheses. Genetic tests at different variants can be correlated, for example because of linkage disequilibrium. A list of different variants is therefore not enough to establish independence.

Definition and Assumptions

Definition: Consider a prespecified family of $m$ hypothesis tests and a target family-wise error rate $\alpha$, where $0<\alpha<1$. The Šidák per-test threshold is $$ c_{\text{S}}=1-(1-\alpha)^{1/m}. $$ Reject a hypothesis when its p-value is at most $c_{\text{S}}$. Under independent tests with valid, continuous, uniformly distributed p-values under their null hypotheses, this threshold gives FWER $\alpha$ when all $m$ hypotheses are null. It gives strong FWER control when the true-null p-values are mutually independent.

A valid p-value under a null hypothesis has probability at most $u$ of being no greater than $u$, for every $u$ between 0 and 1. For an exact continuous test, that probability equals $u$, so the null p-value is uniform on $[0,1]$. A conservative or discrete p-value may have probability less than $u$; in that case the Šidák formula provides an upper bound on the FWER rather than necessarily attaining it exactly.

The family size $m$ must be defined before applying the threshold. In a GWAS, it consists of the hypotheses included in the multiplicity adjustment, according to the analysis plan. The threshold is not calculated using the number of results that happen to look interesting.

Deriving the Šidák Threshold

First suppose all $m$ null hypotheses are true, and each test uses a common cutoff $c$. Under the stated assumptions, each null p-value exceeds $c$ with probability $1-c$. Independence means the probability that all $m$ p-values exceed $c$ is the product of these probabilities. The complement of that event is at least one false rejection.

\begin{aligned} P(\text{at least one false rejection}) &=1-P(\text{all }m\text{ p-values exceed }c)\\ &=1-(1-c)^m. \end{aligned}

Set this probability equal to the target $\alpha$, then solve for $c$. The solution is the Šidák threshold.

1-(1-c)^m=\alpha \quad\Longrightarrow\quad (1-c)^m=1-\alpha \quad\Longrightarrow\quad c=1-(1-\alpha)^{1/m}.

For strong control, some hypotheses may be false. Only rejections of true null hypotheses count toward the FWER. If there are $m_0$ true nulls and their p-values are independent and valid, the probability of at least one false rejection at cutoff $c_{\text{S}}$ is at most $$ 1-(1-c_{\text{S}})^{m_0} \leq 1-(1-c_{\text{S}})^m =\alpha, $$ because $m_0\leq m$. Thus using the full family size controls the FWER even when not all hypotheses are null. With exact continuous null p-values, the first probability is equal to the expression involving $m_0$; conservative null p-values can make it smaller.

Worked Example: Deriving a Threshold for Four Independent Tests

Suppose four hypothetical association tests are independent, their null p-values are exact and continuous, and the target FWER is $\alpha=0.05$. Calculate the Šidák threshold and compare the exact FWER at the Bonferroni cutoff.

Šidák threshold. Here $m=4$, so $$ c_{\text{S}}=1-(1-\alpha)^{1/m} =1-(1-0.05)^{1/4} =1-0.95^{1/4}. $$ The fourth root is approximately $0.9872585449$. Therefore $$ c_{\text{S}}\approx 1-0.9872585449=0.0127414551. $$ A p-value at or below approximately $0.01274$ meets the Šidák cutoff.

Check the family-wise error. At this cutoff, independence gives $$ 1-(1-c_{\text{S}})^4 =1-(0.9872585449)^4 \approx 0.05000. $$ The calculation attains the target, up to rounding.

Bonferroni uses $c_{\text{B}}=\alpha/m=0.05/4=0.0125$. At that cutoff, the exact FWER under these independent, exact null tests is $$ 1-(1-0.0125)^4 =1-(0.9875)^4 \approx 1-0.9509297119 =0.0490702881. $$ Both procedures control the FWER at 0.05, but Bonferroni is more conservative here: its actual FWER is about $0.04907$, while the Šidák cutoff reaches $0.05$.

Šidák-Adjusted P-Values

A threshold answers whether a p-value crosses a chosen family-wise error target. An adjusted p-value expresses the same decision in a way that can be compared with any chosen target $\alpha$. For an observed raw p-value $p$ from one test in a family of $m$ independent tests, the Šidák-adjusted p-value is

p_{\text{S, adj}}=1-(1-p)^m.

Under the all-null model with independent uniform p-values, this is the probability that at least one of the $m$ p-values is no greater than $p$. The adjusted p-value is also a convenient family-wise decision measure: comparing it with $\alpha$ gives the same decision as comparing the raw p-value with $c_{\text{S}}$. In particular, $$ p_{\text{S, adj}}\leq\alpha \quad\Longleftrightarrow\quad p\leq 1-(1-\alpha)^{1/m}=c_{\text{S}}. $$

For strong control under independence, the same comparison remains valid when some hypotheses are false: a raw p-value passing this full-family threshold is a rejection, and the probability of any false rejection among the independent true-null tests is bounded as shown above. The adjusted p-value is not the probability that the tested association is false, nor is it the probability that the null hypothesis is true.

Worked Example: Adjusting a P-Value Across Ten Tests

Suppose a hypothetical study has a prespecified family of $m=10$ independent tests and one test produces a raw p-value of $p=0.01$. Calculate its Šidák-adjusted p-value and decide whether it meets a FWER target of 0.05.

Substitute the raw p-value and family size into the adjustment: $$ p_{\text{S, adj}}=1-(1-p)^m =1-(1-0.01)^{10} =1-(0.99)^{10}. $$ Since $0.99^{10}\approx 0.904382075$, $$ p_{\text{S, adj}}\approx 1-0.904382075=0.095617925. $$ The adjusted value is approximately $0.09562$, which exceeds $0.05$; the result does not meet the 5% FWER criterion.

As a threshold check, the Šidák cutoff for this family is $$ c_{\text{S}}=1-0.95^{1/10}\approx 0.0051162. $$ The raw p-value $0.01$ is greater than $0.0051162$, giving the same conclusion. Under the independence assumptions, the adjusted value summarizes the multiplicity penalty for this ten-test family; it does not alter the raw p-value itself.

Comparison with Bonferroni

Bonferroni assigns each of $m$ tests the cutoff $\alpha/m$. Its validity follows from the union bound, which says that the probability of at least one event is no greater than the sum of the individual probabilities. Under independent tests, Šidák instead uses the exact probability of no rejection. For $m>1$, $0<\alpha<1$, its threshold is slightly larger than the Bonferroni threshold: $$ 1-(1-\alpha)^{1/m}>\frac{\alpha}{m}. $$ One way to see the strict inequality is that at the Bonferroni cutoff the independent-test FWER is $1-(1-\alpha/m)^m$, which is strictly less than $\alpha$ for $m>1$. Increasing the common cutoff to the Šidák value makes this FWER exactly $\alpha$. For $m=1$, both thresholds equal $\alpha$.

The same contrast appears in adjusted p-values. Bonferroni uses $\min(1,mp)$, while Šidák uses $1-(1-p)^m$. For $0<p<1$ and $m>1$, the Šidák adjusted value is no greater than the Bonferroni-adjusted value. The difference is usually small when the target error rate and per-test probabilities are small. A Taylor expansion around $\alpha=0$ makes the leading difference visible: $$ c_{\text{S}} =\frac{\alpha}{m} +\frac{m-1}{2m^2}\alpha^2 +O(\alpha^3). $$ Thus the thresholds agree to first order in $\alpha$, with the Šidák cutoff slightly larger beyond that first-order approximation.

Worked Example: Comparing Decisions in a Five-Test Family

Consider five hypothetical independent association tests with raw p-values $0.002,\ 0.010,\ 0.012,\ 0.040,$ and $0.200$. Assume valid continuous null p-values and a target FWER of $0.05$. Find the Šidák cutoff and determine which results pass it.

The threshold is $$ c_{\text{S}}=1-0.95^{1/5} \approx 1-0.989793779 =0.010206221. $$ Thus the first two raw p-values, $0.002$ and $0.010$, are at or below the threshold. The other three, $0.012,\ 0.040,$ and $0.200$, exceed it.

For the result with raw p-value $0.010$, its adjusted value is $$ 1-(1-0.010)^5 =1-(0.99)^5 =1-0.9509900499 =0.0490099501. $$ It is below $0.05$, consistent with its passing the threshold. For the $0.002$ result, the adjustment is $$ 1-(0.998)^5\approx 1-0.9900399201=0.0099600799, $$ also below $0.05$. The remaining raw p-values exceed $c_{\text{S}}$, so their adjusted p-values exceed $0.05$ by the threshold-adjustment equivalence. These decisions describe evidence under a multiplicity procedure; they do not establish that any associated variant causes a phenotype.

When to Use It and a Common Pitfall

The Šidák method is useful when the testing family is specified and independence of the relevant null tests is credible. In genetic association analysis, different variants may have correlated genotypes, and resulting test statistics can therefore be dependent. Independence should not be assumed simply because the tests have different variant labels. If independence is not justified, the exact Šidák calculation need not control the FWER at the claimed level; Bonferroni’s union-bound guarantee does not require independence.

Common mistake: “There are $m$ tests, so the exact family-wise probability at a per-test cutoff $c$ is $1-(1-c)^m$, regardless of the relationship between tests.”
Correction: The product $(1-c)^m$ is the probability that every test avoids rejection only under independence (and exact null probabilities). Under dependence, that joint probability generally cannot be obtained by multiplying marginal probabilities.
Another mistake: interpreting an adjusted p-value as the probability that a particular null is true. It is a multiplicity-adjusted measure used for a family-wise decision, not a posterior probability about the hypothesis.

Key takeaway: With $m$ independent tests, the Šidák cutoff $1-(1-\alpha)^{1/m}$ makes the probability of at least one false rejection equal to $\alpha$ when all nulls are true. It is slightly less conservative than Bonferroni, but its independence condition must be credible.

Check Your Understanding

Use the assumptions stated in each question, and distinguish the family-wise target from a per-test cutoff.

For $m=8$ independent tests and $\alpha=0.05$, write the Šidák threshold formula. What probability event is used to derive it?
For independent tests with $m=6$, a raw p-value is $0.01$. Write the Šidák-adjusted p-value formula and explain what comparing it with $\alpha=0.05$ tells you.
Why does independence allow the Šidák threshold to be slightly larger than the Bonferroni threshold while preserving the same FWER target?
A GWAS tests variants that may be correlated because of linkage disequilibrium. Why is having different variant identifiers not sufficient to justify the Šidák calculation?
Explain why a Šidák-adjusted p-value is not the probability that the association is false.