Familywise Coverage for Multivariate Confidence Statements

Why Coverage Must Be Considered as a Family

A multivariate analysis may report several intervals for several mean contrasts. Each interval can look reassuring on its own, but the collection makes a stronger claim: that all of the target values are covered at once. The probability of this joint event is not determined by the confidence level of any one interval alone. It also depends on how the interval errors are related.

The distinction matters whenever a reader will interpret a group of statements together. For example, a report might give intervals for several treatment differences and describe all of them as supported by the analysis. The relevant question is then not merely how often one interval succeeds, but how often the entire family succeeds.

Definition: Let $\psi_1,\ldots,\psi_m$ be a finite family of mean contrasts, and let $I_1,\ldots,I_m$ be random confidence intervals for them. The simultaneous coverage event is $$ \{\psi_1\in I_1,\ldots,\psi_m\in I_m\} = \bigcap_{j=1}^m\{\psi_j\in I_j\}. $$ The procedure has simultaneous coverage at least $1-\alpha$ over a parameter space $\Theta$ if $$ \inf_{\theta\in\Theta} P_\theta\!\left(\psi_j\in I_j\text{ for every }j=1,\ldots,m\right) \geq 1-\alpha. $$ The familywise error rate (FWER) at $\theta$ is the probability that at least one target value is missed: $$ \operatorname{FWER}(\theta) = P_\theta\!\left(\text{there exists }j\text{ such that }\psi_j\notin I_j\right). $$

The probability is over repeated samples under the specified model. The parameter values are fixed; it is the intervals that vary from sample to sample. Controlling FWER at level $\alpha$ means that the probability of one or more misses is at most $\alpha$, uniformly over the parameter space. Equivalently, the family has simultaneous coverage at least $1-\alpha$.

Simultaneous Coverage and Marginal Coverage

A marginal coverage probability refers to one interval at a time: $P_\theta(\psi_j\in I_j)$. Simultaneous coverage instead concerns the intersection of all the coverage events. Since that intersection is contained in each individual event, simultaneous coverage cannot exceed any of the marginal coverage probabilities. The reverse inference does not hold: strong marginal coverage does not by itself guarantee strong family coverage.

Result: If a family of intervals has simultaneous coverage at least $1-\alpha$, then every interval has marginal coverage at least $1-\alpha$. However, marginal coverage of at least $1-\alpha$ for every interval does not, in general, imply simultaneous coverage of at least $1-\alpha$.

Proof. Write $A_j=\{\psi_j\in I_j\}$. For each $j$, the event $\bigcap_{k=1}^m A_k$ is a subset of $A_j$. Therefore $$ P_\theta\!\left(\bigcap_{k=1}^m A_k\right) \leq P_\theta(A_j). $$ If the left-hand side is at least $1-\alpha$, then $P_\theta(A_j)\geq1-\alpha$ for every $j$. The converse fails in general, as the next example demonstrates. $\square$

Worked Example: Two Marginally 95% Intervals

Suppose two estimates have independent standard normal errors: $\hat\psi_1=\psi_1+Z_1$ and $\hat\psi_2=\psi_2+Z_2$, where $Z_1,Z_2$ are independent $N(0,1)$ variables. Use the intervals $I_j=[\hat\psi_j-1.96,\hat\psi_j+1.96]$. What are their marginal and simultaneous coverage probabilities?

For either coordinate, coverage is the event $|Z_j|\leq1.96$. Thus its marginal coverage probability is $$ P(|Z_j|\leq1.96)=2\Phi(1.96)-1\approx0.9500, $$ where $\Phi$ is the standard normal distribution function. Because the errors are independent, the two coverage events are independent. Hence $$ P(\psi_1\in I_1,\ \psi_2\in I_2) \approx(0.9500)^2=0.9025. $$ The probability that at least one interval misses is approximately $1-0.9025=0.0975$. So although each interval is marginally about 95% in the usual sense, the pair covers both values only about 90.25% of the time.

This calculation also shows why the number of reported statements matters. A family can have a higher chance of at least one miss than any single interval has of a miss. But it is not enough to know the number of intervals: dependence among their coverage events also affects the joint probability.

Worked Example: The Same Marginal Coverage, Different Dependence

Keep two target values $\psi_1,\psi_2$, but now suppose $\hat\psi_1=\psi_1+Z$ and $\hat\psi_2=\psi_2+Z$, with a single $Z\sim N(0,1)$. Use the same interval form $I_j=[\hat\psi_j-1.96,\hat\psi_j+1.96]$.

For each interval separately, the coverage event is $|Z|\leq1.96$, so each marginal coverage is approximately 0.95. In this model the two events are identical, not independent. Both intervals cover their respective targets precisely when $|Z|\leq1.96$. Therefore $$ P(\psi_1\in I_1,\ \psi_2\in I_2) =P(|Z|\leq1.96)\approx0.95. $$ The FWER is about $1-0.95=0.05$, rather than the 0.0975 found for independent errors. The difference comes entirely from dependence: one sample fluctuation either makes both intervals cover or makes both miss.

The example uses perfectly shared error to make the calculation transparent. In applications, the dependence may be more complicated, and the coverage events need not be identical or independent. The general definition still applies: evaluate the probability that every target is covered under the joint sampling distribution.

A Set of Coordinate Intervals and Its Familywise Meaning

For coordinate intervals, the targets are the separate mean coordinates. The same interpretation applies if the targets are other prespecified mean contrasts: the familywise statement concerns the event that every reported interval contains its own target. The following example uses independent normal errors so that the joint probability can be calculated directly.

Worked Example: Three Coordinate Intervals With at Least 95% Family Coverage

Suppose three estimated mean coordinates have known standard errors $0.5$, $0.4$, and $1.0$. Assume their standardized errors are independent standard normal variables. Use intervals with half-width $2.576$ times the corresponding standard error. For an observed estimate $(10,-2,4)$, calculate the intervals and the familywise coverage.

The half-widths are $2.576(0.5)=1.288$, $2.576(0.4)=1.0304$, and $2.576(1.0)=2.576$. The resulting intervals are $$ [10-1.288,\ 10+1.288]=[8.712,\ 11.288], $$ $$ [-2-1.0304,\ -2+1.0304]=[-3.0304,\ -0.9696], $$ and $$ [4-2.576,\ 4+2.576]=[1.424,\ 6.576]. $$ For each coordinate, the standardized error lies between $-2.576$ and $2.576$ with probability approximately $0.99$. Independence gives the probability that all three intervals cover their respective coordinates: $$ 0.99^3=0.970299. $$ Thus the family has about 97.03% simultaneous coverage under the stated model, and its FWER is $1-0.970299=0.029701$, about 2.97%. This exceeds a 95% familywise coverage target. The observed endpoints describe one sample; the coverage calculation describes the procedure over repeated samples.

This example illustrates how to interpret a familywise guarantee, not a general recipe for constructing intervals. In a different problem, errors may be correlated, standard errors may be estimated, or the family may contain different contrasts. The joint coverage claim must match the actual model and the actual interval procedure.

What a Familywise Claim Does—and Does Not—Say

A familywise coverage statement is about the whole specified family. If a procedure has 95% simultaneous coverage, then in repeated sampling the probability that all family members are covered is at least 0.95. It does not mean that every interval has exactly 95% marginal coverage; some may have more. Nor does it say that, after observing a particular set of intervals, there is a 95% probability that all fixed parameter values lie inside them.

The family also needs to be clear. If intervals are selected or highlighted after inspecting the data, a guarantee for a different, prespecified set may not apply to the selected collection. A reported familywise level should refer to the collection of statements that readers are meant to consider together.

Key takeaway: Marginal coverage describes one interval; simultaneous coverage describes the intersection of all coverage events. Familywise error is the probability of at least one miss. Dependence among interval errors determines how marginal performance translates into family performance.

Common Mistake

Wrong reasoning: “Each of four intervals has 95% coverage, so the probability that all four cover is 95%.” This treats four marginal probabilities as if they automatically specified their joint probability.

Correction: The joint probability depends on the relationship among the coverage events. If four events were independent and each had probability 0.95, their intersection would have probability $0.95^4\approx0.8145$, not 0.95. With dependence, the intersection could be different. A familywise claim must therefore be justified for the joint event, not inferred from marginal coverage alone.

Check Your Understanding

Answer these questions by focusing on the event that all intervals cover.

In your own notation, write the simultaneous coverage event for a finite family of $m$ intervals.
How is the familywise error event related to the simultaneous coverage event?
Why does simultaneous coverage of at least 95% imply at least 95% marginal coverage for each interval?
Two independent intervals each have 90% coverage. What is their joint coverage if their coverage events are independent?
Can marginal coverage probabilities alone determine the probability that all intervals cover? Explain.