Sidak Adjustment and Its Independence Condition

Why the Per-Contrast Level Depends on the Family

A simultaneous confidence procedure reports several intervals and asks for all of them to cover their corresponding parameters at once. If each interval is built at an individual error level $\alpha_0$, the probability that at least one interval misses generally exceeds $\alpha_0$. The familywise error rate is the probability of at least one miss anywhere in the family.

Bonferroni chooses each individual error level small enough to control that probability regardless of dependence. The Šidák adjustment uses a different calculation: it chooses a common per-contrast level so that the familywise error is exactly the target when the interval-coverage events are independent. That condition is substantive. The fact that each interval has a correctly calibrated marginal error rate does not by itself make the Šidák familywise calculation valid.

Definition: For $m$ intervals, let $C_j$ be the event that interval $j$ covers its target parameter, for $j=1,\ldots,m$. The familywise coverage event is $\bigcap_{j=1}^m C_j$, and its complement is the event of at least one noncoverage. The familywise error rate is $P\bigl((\bigcap_{j=1}^m C_j)^c\bigr)$.

Deriving the Šidák Adjustment

Suppose every interval has the same marginal coverage probability $1-\alpha_0$, and suppose the $m$ coverage events $C_1,\ldots,C_m$ are mutually independent. Independence gives the probability of simultaneous coverage directly:

P\left(\bigcap_{j=1}^m C_j\right) =\prod_{j=1}^m P(C_j) =(1-\alpha_0)^m.

To make this probability equal to a desired familywise coverage level $1-\alpha$, solve $(1-\alpha_0)^m=1-\alpha$. The resulting per-contrast error level is the Šidák level.

Definition: For target familywise error rate $0<\alpha<1$ and $m$ intervals, the Šidák per-contrast error level is $$ \alpha_{\mathrm{Sidak}}=1-(1-\alpha)^{1/m}. $$ If the interval events are mutually independent and each has noncoverage probability $\alpha_{\mathrm{Sidak}}$, their simultaneous coverage probability is exactly $1-\alpha$.

For a two-sided interval based on a standard normal statistic, the per-contrast error $\alpha_{\mathrm{Sidak}}$ is split evenly between the two tails. The critical value is $z_{1-\alpha_{\mathrm{Sidak}}/2}$, where $z_q$ is the $q$-quantile of the standard normal distribution. For a statistic with a different marginal distribution, use that distribution's quantile at the corresponding tail probability. The adjustment determines the required marginal tail probability; it does not determine the statistic's distribution.

Worked Example: Two Independent Two-Sided Normal Intervals

Suppose two independent standard normal contrast statistics are used to form two-sided intervals, and the desired familywise error rate is $\alpha=0.05$. Find the Šidák error level and compare the critical value and interval margin with Bonferroni when each contrast has standard error $0.5$.

Šidák allocation. With $m=2$, $$ \alpha_{\mathrm{Sidak}}=1-(1-0.05)^{1/2} =1-\sqrt{0.95} \approx 0.0253206. $$ The two-sided upper-tail quantile is therefore $$ 1-\frac{\alpha_{\mathrm{Sidak}}}{2} \approx 1-0.0126603 =0.9873397. $$ Thus $z_{0.9873397}\approx 2.2365$ (about $2.237$ to three decimal places). The margin of error for a contrast with standard error $0.5$ is $$ 2.2365(0.5)=1.11825\approx 1.1183. $$

Bonferroni comparison. Bonferroni assigns $0.05/2=0.025$ error to each contrast. Each two-sided interval therefore uses quantile $z_{0.9875}\approx 2.2414$, giving margin $$ 2.2414(0.5)=1.1207. $$ The Šidák margin is about $1.1207-1.1183=0.0024$ units shorter on each side. This small difference is the consequence of using a slightly larger per-contrast error level under the independence calculation.

Why Bonferroni Is More Conservative

Bonferroni does not multiply coverage probabilities. It bounds the probability of a union of noncoverage events by the sum of their probabilities. This works without independence, but the bound can be larger than the actual familywise error.

Theorem: If each of $m$ intervals has marginal noncoverage probability at most $\alpha/m$, then the familywise error rate is at most $\alpha$, for any dependence structure among the intervals.

Proof. Let $C_j^c$ denote noncoverage of interval $j$. The familywise error event is $\bigcup_{j=1}^m C_j^c$. The union bound gives $$ P\left(\bigcup_{j=1}^m C_j^c\right) \leq \sum_{j=1}^m P(C_j^c) \leq \sum_{j=1}^m \frac{\alpha}{m} =\alpha. $$ No independence assumption appears in this argument. $\square$

For $m>1$, the Šidák level is slightly larger than the Bonferroni level $\alpha/m$. To verify the direction, the function $x\mapsto x^{1/m}$ is concave for $m>1$. Its tangent at $x=1$ lies above the function, so for $0<\alpha<1$, $$ (1-\alpha)^{1/m}\leq 1-\frac{\alpha}{m}. $$ Subtracting both sides from one yields $$ \alpha_{\mathrm{Sidak}}=1-(1-\alpha)^{1/m}\geq \frac{\alpha}{m}. $$ For $m>1$ and $0<\alpha<1$, the inequality is strict. A larger marginal error level means a smaller two-sided critical value and, all else equal, narrower intervals. This comparison concerns the allocations; it does not make Šidák valid without its independence condition.

Worked Example: Comparing Allocations for Five Intervals

For $m=5$ intervals and target familywise error $\alpha=0.05$, compare the two per-contrast levels.

Bonferroni: $$ \alpha_{\mathrm{Bonf}}=\frac{0.05}{5}=0.010000. $$

Šidák: $$ \alpha_{\mathrm{Sidak}}=1-0.95^{1/5} \approx 1-0.9897938 =0.0102062. $$

The Šidák allocation is about $0.0002062$ larger per contrast. If the intervals are two-sided normal intervals, the corresponding upper-tail probabilities are about $0.0051031$ for Šidák and $0.0050000$ for Bonferroni. Thus the Šidák critical value is slightly lower. The exact familywise error calculation for Šidák requires independence of the five coverage events; the Bonferroni guarantee does not.

What Independence Means for Contrasts

An interval event is determined by its contrast statistic: for example, a two-sided interval covers its target when the absolute value of a standardized statistic does not exceed the chosen critical value. If the contrast statistics are mutually independent, then events determined separately by each statistic are also mutually independent. The Šidák product calculation is then justified, provided each statistic is calibrated to give the specified marginal coverage.

In a multivariate normal model with known covariance, a vector of linear contrasts is jointly normal. Two such contrasts are independent if their covariance is zero. For contrast vectors $c_i$ and $c_j$ applied to a random vector with covariance matrix $\Sigma$, that covariance is $c_i^\mathsf{T}\Sigma c_j$. Consequently, checking whether contrast vectors look geometrically distinct or orthogonal in ordinary coordinates is not enough; their covariance under the model is what matters.

With estimated covariance, additional care is needed. Contrast numerators can be uncorrelated, yet studentized statistics can share estimated scale quantities. Independence of the statistics—or directly of the interval events—must be established for the actual procedure, not inferred merely from the contrast definitions.

Worked Example: Duplicate Contrasts Break the Product Calculation

Consider two reported contrasts that are identical, so their standardized statistics satisfy $T_1=T_2$ exactly. Suppose the target familywise error is $0.05$. What does the Šidák calculation predict, and what is the actual familywise error?

For two intervals, Šidák assigns $$ \alpha_{\mathrm{Sidak}}=1-\sqrt{0.95}\approx 0.0253206 $$ to each contrast. But because $T_1=T_2$, the two intervals have the same coverage event: $C_1=C_2$. Therefore $$ P(C_1\cap C_2)=P(C_1)=1-\alpha_{\mathrm{Sidak}} \approx 0.9746794, $$ not $(1-\alpha_{\mathrm{Sidak}})^2=0.95$. The actual familywise error is $0.0253206$, not $0.05$. This particular dependence makes the Šidák procedure conservative; the important point is that its claimed exactness from multiplying two probabilities does not apply.

Worked Example: Checking Covariance Between Two Contrasts

Let $X$ be a bivariate normal random vector with mean vector $\mu$ and covariance matrix $$ \Sigma=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}. $$ Consider contrast vectors $c_1=(1,0)^\mathsf{T}$ and $c_2=(1,1)^\mathsf{T}$. Determine whether the two contrast statistics can be treated as independent in the known-covariance normal model.

Their contrast values are $c_1^\mathsf{T}X=X_1$ and $c_2^\mathsf{T}X=X_1+X_2$. Their variances are $$ \operatorname{Var}(c_1^\mathsf{T}X)=c_1^\mathsf{T}\Sigma c_1=1, \qquad \operatorname{Var}(c_2^\mathsf{T}X)=c_2^\mathsf{T}\Sigma c_2=1+0.6+0.6+1=3.2. $$ Their covariance is $$ \operatorname{Cov}(c_1^\mathsf{T}X,c_2^\mathsf{T}X) =c_1^\mathsf{T}\Sigma c_2=1+0.6=1.6. $$ The correlation is therefore $$ \frac{1.6}{\sqrt{1(3.2)}}\approx 0.8944. $$ Since the pair is jointly normal and has nonzero covariance, the two contrast statistics are not independent. Thus this covariance calculation does not support the exact Šidák multiplication of their interval-coverage probabilities. A dependence-aware calculation or a procedure with a dependence-robust guarantee is needed instead.

Common Mistake and Practical Interpretation

A common mistake is to compute $1-(1-\alpha)^{1/m}$ for any family of contrasts and then call the resulting intervals an exact $1-\alpha$ simultaneous procedure. The formula is not a universal replacement for Bonferroni: it comes from setting the product of independent coverage probabilities equal to the desired coverage. Correlation among contrast statistics removes that justification. It does not follow that Šidák must always be anti-conservative under dependence; as the duplicate-contrast example shows, dependence can make it conservative. Rather, without the required independence (or a separate theorem establishing an appropriate inequality), the exact product calculation cannot be asserted.

In practice, state the dependence assumption alongside the adjustment. If interval events are mutually independent and have equal marginal noncoverage $\alpha_{\mathrm{Sidak}}$, Šidák gives exact familywise coverage. If dependence is present or not established, Bonferroni still gives familywise error at most $\alpha$ when each marginal error is at most $\alpha/m$. More refined methods can use the joint distribution of the contrast statistics, but their calibration requires that joint dependence structure.

Key takeaway: Šidák chooses the per-contrast level by solving $(1-\alpha_0)^m=1-\alpha$. Its exactness rests on independence of the interval-coverage events. Bonferroni instead uses the union bound and controls familywise error without that assumption.

Check Your Understanding

Use the event-based derivation to answer the following questions.

For $m$ independent intervals with common marginal noncoverage $\alpha_0$, write the equation that determines $\alpha_0$ for target familywise error $\alpha$.
For a two-sided normal interval, how is the Šidák per-contrast error level converted to the upper-tail probability used to choose the critical value?
Why does the union bound establish Bonferroni control even when the contrast statistics are correlated?
For two identical contrast statistics, explain why their interval events cannot be multiplied as if independent.
In a jointly normal model, what covariance calculation can be used to assess independence of two linear contrasts when the covariance matrix is known?