False Coverage Rates for Selected Treatment Effects

Why the Error Criterion Must Match the Report

In a multi-arm multi-stage (MAMS) trial, the arms for which treatment-effect intervals are reported may be chosen using interim or final data. The reported set is therefore random: a different repetition of the trial could select different arms and produce a different number of intervals. An error criterion for these reports should account for that selection. One relevant criterion asks not whether every reported interval is correct, but what fraction of the reported intervals fail to cover their treatment effects on average.

This criterion is useful when the scientific report emphasizes a collection of selected treatment effects. It is not a substitute for a guarantee that all reported intervals cover simultaneously. Those two guarantees answer different questions, and their distinction is especially important when the number of selected arms varies.

Definition of the False Coverage Rate

Let there be $K$ candidate treatment arms, with true treatment effects $\theta_1,\ldots,\theta_K$. The trial’s complete data and selection rule produce a random selected set $S\subseteq\{1,\ldots,K\}$. For each selected arm $i$, let $C_i$ be the reported interval for $\theta_i$. Define $R=|S|$, the number of selected arms, and let $V$ be the number of selected arms whose intervals miss their true effects.

Definition: The false coverage proportion is $V/R$ when $R>0$, and is defined to be 0 when $R=0$. The false coverage rate (FCR) is its expectation over repetitions of the entire trial, including data collection, interim decisions, arm selection, and interval construction: $$ \operatorname{FCR}=E\!\left[\frac{V}{\max(R,1)}\right]. $$

The expectation is over repeated trials with the same treatment effects, design, and analysis rules. It is not a probability that a particular interval is false, and it is not generally the expected number of misses divided by the expected number selected. In particular, because $R$ is random, $E[V/R]$ (with the zero convention above) need not equal $E[V]/E[R]$.

An interval miss means $\theta_i\notin C_i$, regardless of whether the estimated effect is too high or too low. The FCR concerns only the intervals that are reported for selected arms; it does not count misses among unselected arms.

Worked Example: The False Coverage Proportion in One Trial

Suppose a hypothetical MAMS trial has six candidate arms and selects arms 1, 3, 5, and 6. Their reported intervals and fixed true effects are as follows.

Selected arm	True effect $\theta_i$	Reported interval $C_i$	Covers?
1	1.0	[0.2, 2.0]	Yes
3	0.5	[0.1, 1.4]	Yes
5	1.2	[1.3, 2.1]	No
6	0.7	[0.1, 1.5]	Yes

Here $R=4$ and $V=1$, because the true effect for arm 5 is below its interval. Thus the false coverage proportion in this one hypothetical trial is $$ \frac{V}{R}=\frac{1}{4}=0.25. $$ This realized proportion is not itself an estimate of the design’s FCR unless it is combined with an appropriate set of repeated trials. It records what happened in this particular realization.

How FCR Differs From Simultaneous Coverage

For the random selected set, simultaneous coverage asks whether every reported interval covers its corresponding effect. Its probability is $P(V=0)$. Equivalently, the familywise noncoverage probability among selected arms is $P(V\geq 1)$, the probability that at least one selected interval misses. Since simultaneous coverage fails whenever $V\geq 1$, its probability is $1-P(V\geq1)$.

The FCR instead averages the fraction of selected intervals that miss. A trial with one miss among many selected intervals contributes a small false coverage proportion, even though simultaneous coverage fails for that trial. A trial with one miss among two selected intervals contributes a larger proportion.

0\leq \frac{V}{\max(R,1)}\leq \mathbf{1}(V\geq1) \quad\Longrightarrow\quad \operatorname{FCR}\leq P(V\geq1).

The inequality follows because the false coverage proportion is zero when there are no misses, and cannot exceed one when there is at least one miss. Taking expectations gives the result. Therefore, a procedure that guarantees simultaneous coverage of the selected intervals with probability at least $1-\alpha$ also guarantees $\operatorname{FCR}\leq\alpha$. The reverse implication does not hold: a small expected fraction of misses does not ensure that the probability of any miss is small.

Worked Example: A Low FCR With Frequent Simultaneous Failure

Consider a hypothetical repeated-trial pattern in which exactly five arms are selected in each of 100 trials. In 20 trials, exactly one of the five intervals misses; in the other 80 trials, all five cover.

The false coverage proportion is $1/5=0.20$ in each of the 20 trials with a miss, and 0 in the other trials. The estimated long-run FCR for this specified pattern is $$ \operatorname{FCR} =\frac{20(0.20)+80(0)}{100} =0.04. $$ The simultaneous coverage failure proportion is instead $20/100=0.20$, so simultaneous coverage is $0.80$. Thus, in this example the FCR is 4%, while the probability of at least one selected interval missing is 20%. A statement that the FCR is at most 5% would not justify saying that all selected intervals jointly cover with probability at least 95%.

Controlling FCR and Choosing a Guarantee

The coverage target should reflect the intended interpretation. If the goal is to limit the expected proportion of incorrect intervals among the reported arms, an FCR target $q$ is relevant. If the goal is to make it unlikely that even one reported interval is incorrect, simultaneous coverage or familywise error control is the more direct target.

One conservative route to FCR control is to construct intervals that cover all $K$ candidate effects simultaneously. For example, if each of $K$ intervals has noncoverage probability at most $\alpha/K$, then the union bound gives probability at most $\alpha$ that any candidate interval misses. Any selected subset is covered whenever all $K$ intervals cover, so the selected-set FCR is also at most $\alpha$. This route does not require independence between arms, but can be conservative.

A less conservative selected-count adjustment is available in a common setting. Suppose armwise estimates and test statistics are independent across arms, each interval is valid under its specified marginal model, and selection is based on armwise significance tests. A standard FCR construction forms each selected arm’s interval at confidence level $1-qR/K$, where $R$ is the number selected. Under these conditions, the resulting intervals control FCR at $q$. This is an FCR guarantee, not a simultaneous-coverage guarantee at level $1-q$. Dependence across arms, a different selection rule, or a change in the analysis after observing results can invalidate the guarantee unless the procedure is justified for that setting.

Worked Example: Applying a Selected-Count Adjustment

Suppose a hypothetical trial has $K=5$ independent armwise estimates $\hat{\theta}_i\sim N(\theta_i,1^2)$, with known standard error 1 for each arm. Arms are selected when their two-sided test against zero has $|Z_i|>1.96$. In one realization, two arms are selected, with estimates 2.5 and 2.8, so $R=2$. For an FCR target $q=0.05$, the selected-count adjustment uses per-interval noncoverage level $$ \frac{qR}{K}=\frac{0.05(2)}{5}=0.02. $$ Each interval therefore has nominal confidence level $1-0.02=0.98$. The two-sided standard-normal critical value is approximately $z_{0.99}=2.326$.

For the arm with estimate 2.5, the interval is $$ 2.5\pm 2.326(1)=[0.174,\;4.826]. $$ For the arm with estimate 2.8, it is $$ 2.8\pm 2.326(1)=[0.474,\;5.126]. $$ The arithmetic gives the intervals for this realization; it does not reveal whether either interval contains its unknown true effect. Under the stated independence, model, and selection conditions, the procedure’s repeated-trial FCR is controlled at 0.05. It does not promise that both intervals cover together with probability 0.95.

Diagnostics, Limitations, and Common Pitfalls

FCR is a repeated-procedure property, so diagnostics should reproduce the entire selection-and-reporting process. In a design simulation, record $R$ and $V$ in each replicate, calculate $V/\max(R,1)$, and average these proportions across replicates. Also record $P(V\geq1)$ separately if simultaneous reliability matters. Reporting only the average number of misses, or only the percentage of intervals that cover without accounting for selection and the random denominator, does not establish FCR control.

Confusing the target with simultaneous coverage. An FCR target of 0.05 means an expected false coverage proportion of at most 5%; it does not mean a 95% probability that every reported interval covers.
Dividing expected counts. $E[V]/E[R]$ is not generally the FCR, which is the expectation of the replicate-specific proportion.
Using one trial to claim a long-run rate. A realized proportion describes one trial, not the repeated-trial expectation.
Ignoring the selection mechanism. Intervals calibrated as if the reported arms had been fixed in advance may not retain the desired error guarantee after data-dependent selection.
Assuming independence without checking the design. Shared controls, common interim decisions, and correlated outcomes can induce dependence between armwise statistics. A guarantee established under independence should not be carried over automatically.

FCR control also has limitations as a summary. It weights each trial’s proportion equally, not each selected interval equally across all repetitions. It can be small when many arms are selected even if at least one interval often misses. It also does not promise a particular coverage probability conditional on the exact selected set or on a particular selection path. When those conditional guarantees are needed, they must be specified and assessed separately.

Key takeaway: FCR is the expected fraction of selected-arm intervals that fail to cover. It is bounded above by the probability of any selected interval failing, so simultaneous coverage control implies FCR control; FCR control alone does not imply simultaneous coverage.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In one trial, six arms are selected and two reported intervals miss. What is the false coverage proportion for that trial?
State the event whose probability is the simultaneous coverage failure probability among selected arms. How does it relate to the false coverage proportion?
Across 50 hypothetical trials, exactly four trials have one miss among four selected intervals, and all other trials have no misses. Calculate the empirical average false coverage proportion and the empirical simultaneous coverage failure proportion.
Why does an FCR bound of 0.05 not imply at least 95% simultaneous coverage?
What quantity should a design simulation average to estimate FCR when the number of selected arms varies?

Solutions. 1. The false coverage proportion is $2/6=0.333$ (to three significant figures). 2. Simultaneous coverage fails on the event $V\geq1$; the false coverage proportion is zero when $V=0$, and at most 1 when $V\geq1$, so its expectation cannot exceed $P(V\geq1)$. 3. Each of the four trials with a miss contributes $1/4$, and the remaining 46 contribute 0. The empirical FCR is $[4(1/4)+46(0)]/50=0.020$. Simultaneous coverage fails in four of 50 trials, giving $4/50=0.080$. 4. FCR averages the fraction of misses, whereas simultaneous failure counts any trial with at least one miss as a failure. The fraction can be small even when at least one miss occurs often. 5. For each replicate calculate $V/\max(R,1)$, then average those replicate-specific proportions.