Diagnosing Simultaneous Coverage Failures

What Does a Simultaneous Coverage Failure Mean?

A reported coverage rate below its target does not, by itself, show that a critical value is wrong. The simulation may have evaluated a different contrast family from the one the procedure was designed to cover, used an incorrect covariance matrix, applied a cutoff calibrated for another statistic, or counted separate marginal successes instead of asking whether all intervals covered together. Diagnosis starts by writing down the exact event whose probability is being estimated.

Let $\hat\theta$ estimate a parameter vector $\theta\in\mathbb{R}^p$, and let $C$ be an $m\times p$ matrix whose rows specify the intended contrasts. The target vector is $C\theta$, and the estimated contrast vector is $C\hat\theta$. For intervals $I_1,\ldots,I_m$, simultaneous coverage means that every target contrast is covered in the same repetition.

Definition: The simultaneous coverage probability for the contrast family $C$ is $$ P\bigl(C_j\theta\in I_j\ \text{for every }j=1,\ldots,m\bigr), $$ where $C_j$ is row $j$ of $C$. This is one joint probability, not the collection of $m$ separate marginal coverage probabilities.

For symmetric max-$t$ intervals, let $\Omega$ be the covariance matrix of $C\hat\theta$, and suppose its diagonal entries are positive. Write $D=\operatorname{diag}(\Omega_{11},\ldots,\Omega_{mm})$, and define the standardized contrast errors and their correlation matrix by

Z=D^{-1/2}C(\hat\theta-\theta), \qquad R=D^{-1/2}\Omega D^{-1/2}.

When $\hat\theta\sim N_p(\theta,\Sigma)$ and $\Omega=C\Sigma C^{\mathsf T}$ is known, $Z\sim N_m(0,R)$. A max-$t$ procedure with cutoff $c$ uses intervals

C_j\hat\theta\ \pm\ c\sqrt{\Omega_{jj}},\qquad j=1,\ldots,m.

The simultaneous coverage event is therefore $\max_j|Z_j|\le c$. If a contrast family contains linear dependencies, $R$ may be singular; the maximum is still well-defined, and calibration must reflect that dependence rather than assume independent coordinates.

Result: Exact Calibration Under a Gaussian Model

Theorem: Suppose $\hat\theta\sim N_p(\theta,\Sigma)$, the contrast matrix $C$ is fixed, and every contrast has positive variance under $\Omega=C\Sigma C^{\mathsf T}$. Let $c_{1-\alpha}$ be a $1-\alpha$ quantile of $M=\max_{1\le j\le m}|Z_j|$, with $Z\sim N_m(0,R)$. Then the intervals with cutoff $c_{1-\alpha}$ have simultaneous coverage $1-\alpha$, provided the distribution function of $M$ is continuous at that quantile.

Proof. By linear transformation of a multivariate normal vector, $C(\hat\theta-\theta)\sim N_m(0,\Omega)$. Standardizing each component gives $Z\sim N_m(0,R)$. For any $j$, the interval covers its target exactly when

|C_j\hat\theta-C_j\theta|\le c_{1-\alpha}\sqrt{\Omega_{jj}} \quad\Longleftrightarrow\quad |Z_j|\le c_{1-\alpha}.

All $m$ intervals cover exactly when $\max_j|Z_j|\le c_{1-\alpha}$. By the definition of the quantile and the stated continuity condition, this event has probability $1-\alpha$. Thus the result is about the joint distribution of the standardized contrasts; separate marginal quantiles need not give this coverage. $\square$

This result also supplies an “oracle” benchmark for simulation: if the generating covariance is known and the Gaussian model is used, the cutoff obtained from the correct $R$ should achieve the target up to Monte Carlo error. A procedure that estimates its covariance from data needs a different benchmark: the simulation must reproduce that estimation step and use the calibration intended for the resulting statistic.

Worked Examples: Separating the Failure Modes

Worked Example: Marginal Success, Joint Undercoverage

Let the two contrast errors be independent $N(0,1)$ variables, and consider the two intervals centered at their respective estimates. A pointwise 95% normal interval uses $c=1.96$. What is its simultaneous coverage?

Each interval covers its target with probability $2\Phi(1.96)-1$, approximately $0.95$. Independence makes the probability that both cover the product of their marginal probabilities:

P(\text{both cover})= \{2\Phi(1.96)-1\}^2 =0.95^2 =0.9025.

Thus the intervals have 95% marginal coverage but only 90.25% simultaneous coverage. To calibrate the maximum exactly for this independent pair, choose $c$ so that

\{2\Phi(c)-1\}^2=0.95, \qquad c=\Phi^{-1}\!\left(\frac{1+\sqrt{0.95}}{2}\right) \approx 2.236.

With this cutoff, each marginal coverage is $\sqrt{0.95}\approx0.97468$, and the joint coverage is $0.97468^2=0.95$. In 100,000 repetitions, the pointwise procedure has expected joint successes $100{,}000(0.9025)=90{,}250$. Its Monte Carlo standard error is approximately $\sqrt{0.9025(1-0.9025)/100{,}000}=0.00094$. A simulated joint rate near 0.903 is therefore consistent with the model; it is not evidence of a coding defect if the procedure was intended to provide only pointwise intervals.

Worked Example: A Covariance Scale Error

Suppose there are two independent contrast errors. The true errors have distributions $N(0,1)$ and $N(0,4)$, but the procedure uses the identity matrix as its contrast covariance and applies the two-contrast 95% cutoff $c\approx2.236$ from the preceding calculation. Find its actual simultaneous coverage.

The procedure uses half-width $2.236$ for both contrasts. The first contrast covers with probability $2\Phi(2.236)-1\approx0.97468$. For the second, whose standard deviation is 2, the standardized half-width is $2.236/2=1.118$, so its coverage is $2\Phi(1.118)-1\approx0.736$. Independence gives

P(\text{both cover}) \approx 0.97468(0.736) \approx 0.718.

The nominal cutoff was calibrated for two independent, unit-variance standardized errors, but the second contrast was not standardized correctly. If the procedure used the true standard errors, its half-widths would be $2.236$ and $2(2.236)=4.472$; both standardized errors would then have the same coverage, and the joint coverage would be $0.95$. This example isolates covariance scale from critical-value calibration: the cutoff is appropriate for the stated standardized model, while the covariance supplied to the procedure is not.

Worked Example: The Simulation Omits a Contrast

Suppose the intended family has three independent $N(0,1)$ contrast errors, but the simulation code checks only the first two. The code uses the two-contrast 95% cutoff $c\approx2.236$. What rate does it report, and what is the true coverage of the intended family?

The code reports the probability that its two checked intervals both cover: \[ \{2\Phi(c)-1\}^2=0.95. \] The omitted third contrast also has marginal coverage $2\Phi(c)-1=\sqrt{0.95}$. The actual probability that all three intended intervals cover is therefore

\{2\Phi(c)-1\}^3 =(\sqrt{0.95})^3 =0.95^{3/2} \approx0.926.

The simulation can consequently report 95% for the two contrasts it checks while the procedure achieves only about 92.6% for the stated three-contrast family. For this independent family, the correct three-contrast cutoff solves $\{2\Phi(c_3)-1\}^3=0.95$, or $c_3=\Phi^{-1}((1+0.95^{1/3})/2)\approx2.388$. This diagnosis is about the definition and evaluation of the family, not about covariance estimation.

A Second Check: What Bonferroni Can and Cannot Diagnose

A useful conservative cross-check is the Bonferroni guarantee. It does not require independent contrasts, so it can help distinguish an impossible coverage claim from a potentially conservative one.

Result: If each of $m$ intervals has marginal noncoverage probability at most $\alpha/m$, then their simultaneous coverage is at least $1-\alpha$, regardless of dependence.

Proof. Let $A_j$ be the event that interval $j$ fails to cover. Failure of simultaneous coverage is the union $\bigcup_{j=1}^m A_j$. The union bound gives

P\!\left(\bigcup_{j=1}^m A_j\right) \le \sum_{j=1}^m P(A_j) \le m\frac{\alpha}{m} =\alpha.

Taking complements yields simultaneous coverage at least $1-\alpha$. The inequality need not be equality: dependence among contrasts can make this bound conservative. In a diagnostic simulation, a correctly implemented procedure with the stated marginal error bounds should not have joint coverage below this guarantee except for ordinary simulation variation. $\square$

A Layered Diagnostic for Gaussian Simulations

When a simulation appears to under-cover, keep the target family fixed and compare the following versions of the analysis. Each comparison changes one ingredient at a time.

Write the family explicitly.
Record every row of $C$, including signs, reference coordinates, and any redundant contrasts. In each repetition, test whether every intended contrast is covered.

Check the covariance used for standardization.
In a Gaussian simulation, compare the generating covariance of $C\hat\theta$ with the covariance supplied to the procedure. Check variances and correlations, not just whether the matrix has the right dimensions.

Check the cutoff against the statistic.
Verify that the simulated cutoff uses the same maximum, one- or two-sided convention, correlation structure, and covariance-estimation steps as the intervals being evaluated.

Record joint and marginal rates separately.
For each repetition, record one indicator for all contrasts covering, and also record a coverage indicator for each contrast. Compare these rates with their intended targets.

Use an oracle comparison.
For a known-covariance Gaussian model, calibrate with the generating covariance and the correct contrast family. If this version works but the operational version does not, investigate its covariance estimate or its calibration procedure.

Monte Carlo uncertainty should be reported alongside the estimated rate. If $N$ independent repetitions produce joint coverage estimate $\hat p$, its estimated standard error is $\sqrt{\hat p(1-\hat p)/N}$. A small difference from the target may be simulation noise; a discrepancy many standard errors wide deserves investigation. This uncertainty calculation does not repair a misspecified family or covariance—it only quantifies the variability of the estimated coverage rate.

Common Mistakes

Calling marginal coverage “simultaneous.” Checking each contrast separately can produce high marginal rates while the probability that all intervals cover is lower. Compute the all-contrasts indicator in every replicate.
Blaming the critical value before checking standard errors. A correctly tabulated cutoff cannot compensate for contrast variances or correlations that were supplied incorrectly.
Comparing cutoffs for different statistics. A cutoff for a maximum of two-sided standardized contrasts is not interchangeable with a pointwise normal quantile or a one-sided maximum cutoff.
Evaluating a different family from the one reported. A missing contrast, an unintended duplicate, or a different reference coordinate changes the coverage event and may change the calibrated maximum distribution.
Treating a simulated rate as exact. Coverage estimates fluctuate across finite simulations. Report the number of repetitions and Monte Carlo standard error before interpreting a small discrepancy.

Key takeaway: Diagnose simultaneous undercoverage by fixing the coverage event first, then checking the contrast family, covariance standardization, critical-value calibration, and joint scoring. In a Gaussian model with known covariance, the correctly calibrated maximum statistic gives a direct benchmark for that investigation.

Check Your Understanding

Use these questions to test whether you can distinguish the main sources of apparent undercoverage.

Two independent intervals each have 95% marginal coverage. What is their joint coverage, and why is it not 95%?
In a Gaussian max-$t$ procedure, what covariance matrix determines the correlation structure of the standardized contrast errors?
A simulation uses the correct cutoff but underestimates one contrast standard error. Which part of the procedure should be checked first, and why?
Why can a simulation report target coverage for two contrasts while the intended three-contrast family under-covers?
What is the role of an oracle comparison when a procedure estimates its covariance matrix?
How does the union bound provide a conservative check on simultaneous coverage without assuming independent contrasts?