Why the Conditioning Set Matters
An adaptive trial can stop early, continue to a later stage, or select a treatment based on interim data. An interval procedure may perform differently across those outcomes. Its overall coverage across repeated trials can therefore differ from its coverage among trials that followed one particular path. These are distinct probability statements, and the distinction matters whenever an interval is interpreted in the context of the path that produced it.
The key question is not simply whether an interval has “95% coverage.” It is: 95% over which repeated trials? The answer may refer to all trials run under the design, or only to trials that stopped early, continued, or selected a particular treatment. Neither guarantee automatically implies the other.
Both probabilities are evaluated at a specified parameter value and under a specified trial design. A coverage claim is usually approximate in practice, so “95%” often means close to 0.95 over the relevant parameter range, rather than exactly 0.95 at every possible value. Conditioning must also be stated precisely: “given early stopping” is different from “given the exact interim statistic that caused early stopping.”
How Path-Specific Coverage Combines
Suppose the possible adaptation paths \(A_1,\ldots,A_K\) are mutually exclusive and collectively exhaustive: each trial follows exactly one of them. At a fixed \(\theta\), the unconditional coverage is the probability-weighted average of the conditional coverages on those paths.
This identity follows by partitioning the event that the interval covers. The events \(\{\theta\in C(D)\}\cap A_k\) are disjoint, and their union is the coverage event. For each path, the definition of conditional probability gives \(P_\theta(\{\theta\in C(D)\}\cap A_k)=P_\theta(A_k)P_\theta\{\theta\in C(D)\mid A_k\}\). Adding these probabilities proves the identity. A path with probability zero at the specified \(\theta\) contributes zero to the unconditional coverage, and its conditional coverage is not defined.
Worked Example: Conditioning on a Favorable Interim Result
Consider a hypothetical trial summarized by an estimate \(X\) of a treatment effect \(\theta\), measured in points. Assume the standard error is known to be 1 point and, at the fixed value \(\theta=0\), \(Z=X-\theta\) has a standard normal distribution. The reported interval is \(X\pm1.96\) points. The trial proceeds to a selected-treatment path \(A\) when \(Z>1\). Find the interval’s unconditional and conditional coverage.
The normal model and known standard error imply that the interval covers \(\theta\) precisely when \(-1.96\leq Z\leq1.96\). Thus its unconditional coverage is \(P(-1.96\leq Z\leq1.96)=0.9500\). The selection probability is \(P(A)=P(Z>1)=1-\Phi(1)=0.1587\), where \(\Phi\) is the standard normal cumulative distribution function.
On the selected path, coverage requires \(1<Z\leq1.96\). Therefore,
The nominal interval covers in 95.00% of all trials at \(\theta=0\), but in only about 84.25% of trials that pass the favorable interim threshold. The complement path \(A^c\) has probability \(0.8413\). Its conditional coverage is \((0.9500-0.1337)/0.8413=0.9703\), since the joint probability of coverage and \(A\) is 0.1337. Weighting the path-specific results recovers the overall guarantee: \(0.1587(0.8425)+0.8413(0.9703)\approx0.9500\).
The assumptions here are deliberately simple: the standardized estimate is exactly normal, its standard error is known, and the threshold rule is fixed in advance. In an actual adaptive trial, the sampling distribution and interval may be more complicated. The example isolates the conditioning issue: selecting on a statistic related to the estimate can change coverage on the selected path even when unconditional coverage is nominal.
Stopping Paths and Trial-Wide Guarantees
An early-stop decision also partitions repeated trials. The probability of each path can depend on \(\theta\), so the weights in the coverage identity are generally parameter-dependent. A stated trial-wide guarantee should consequently be understood at a fixed value of \(\theta\), or as a claim about a specified range or set of values. One should not substitute a single stopping probability across all parameter values without justification.
Worked Example: An Early-Stopping Rule
Suppose a hypothetical two-stage design has an early-stop path \(E\) and a continuation path \(C\). At a particular treatment effect, exact repeated-sampling calculations give \(P_\theta(E)=0.20\) and \(P_\theta(C)=0.80\). The reported interval covers on 80% of trials following \(E\) and 98.75% of trials following \(C\). Calculate unconditional coverage and interpret the result.
The paths exhaust all possibilities, and their probabilities sum to \(0.20+0.80=1\). Applying the path-weighted identity gives
Thus the interval has 95.00% unconditional coverage at the stated parameter value, even though coverage conditional on early stopping is only 80.00%. The numbers are hypothetical operating characteristics for illustration; the calculation does not establish that a particular two-stage procedure has these properties. In a real design, the path probabilities and conditional coverages must be evaluated under the actual stopping rule and interval procedure.
If the scientific or regulatory interpretation concerns all trials conducted under the design, the unconditional result answers that repeated-trial question. If the interval is being interpreted specifically as an estimate from an early-stopped trial, the 80% conditional result is directly relevant and the 95% overall figure alone would be incomplete.
Selection and the Guarantee for the Reported Result
Selection creates a related issue. If the reported estimate and interval belong to a treatment chosen using interim results, the selection event is part of the path. A guarantee averaged over all possible selections can be materially different from the guarantee for the treatment that was actually selected. The selection probability is not an optional detail in an unconditional calculation: it supplies the weight for that path.
Worked Example: Coverage for a Selected Treatment
In a hypothetical two-treatment design, suppose treatment A is selected with probability 0.25 at a fixed \(\theta\), and treatment B is selected with probability 0.75. The interval for the selected treatment covers its target effect in 80% of trials when A is selected and in 100% of trials when B is selected. Find the overall coverage and explain what the result says about the selected-treatment interval.
The two selection events are exhaustive. The trial-wide coverage is
The overall coverage is 95.00%, but among trials selecting A it is 80.00%. The result does not support describing the interval as having 95% coverage conditional on selecting A. Nor does it show that the 80% figure applies to trials selecting B: that path has the stipulated 100% coverage. The overall statement is valid only with its trial-wide conditioning set made clear.
Choosing and Checking the Relevant Guarantee
The appropriate guarantee depends on the repeated-trial question. Unconditional coverage is relevant when the target is performance of the complete adaptive design, including all possible stopping and selection outcomes. Conditional coverage is relevant when the inferential claim is explicitly restricted to trials that followed a specified path. In applications, both may be useful: the overall value summarizes design-wide performance, while path-specific values show whether an important route through the trial has weak coverage.
Several checks help prevent a misleading claim:
- Specify the event. Name the path precisely, such as “stopped at the interim analysis” or “selected treatment A.” Do not use “conditional” without stating what is conditioned on.
- Check the weights. Confirm that the listed paths are mutually exclusive and exhaustive, and that their probabilities at the parameter value sum to 1.
- Keep the probability model consistent. Path probabilities and conditional coverages must refer to the same parameter value, design, interval procedure, and repeated-sampling distribution.
- Inspect important paths. A high overall coverage can conceal low coverage on a path that is scientifically or operationally important, especially if that path has a modest probability.
- Do not treat realized path probabilities as universal. A stopping probability calculated at one treatment effect need not apply at another.
| Guarantee | Probability statement | Question it answers | Important limitation |
|---|---|---|---|
| Unconditional coverage | \(P_\theta\{\theta\in C(D)\}\) | How often does the full design’s interval cover across repeated trials? | May conceal weak coverage on a particular stopping or selection path. |
| Conditional coverage | \(P_\theta\{\theta\in C(D)\mid A\}\) | How often does the interval cover among trials following path \(A\)? | Can differ sharply across paths and is undefined when \(P_\theta(A)=0\). |
| Path-by-path assessment | All \(P_\theta\{\theta\in C(D)\mid A_k\}\), with path probabilities | How does coverage vary across the adaptation outcomes? | Requires enough detail to evaluate the path-specific operating characteristics. |
A common mistake is to reason, “The interval has 95% unconditional coverage, so after observing the trial’s selected path it still has 95% coverage.” That conclusion does not follow. The unconditional probability averages conditional coverages using path probabilities; one conditional coverage can be lower and another higher while their weighted average is 0.95. The correction is to report the conditioning set and, when the path matters, evaluate coverage for that path rather than infer it from the overall value.
Conditional and unconditional guarantees are not competing definitions of correctness. They answer different questions. Path-specific intervals or other selection-aware procedures can be designed to target conditional behavior, while an interval calibrated for the complete design may target unconditional behavior. Whichever approach is used, the guarantee should match the intended interpretation and should be checked under the actual adaptation mechanism. An overall coverage number alone cannot establish that every path performs well.
Check Your Understanding
Use the stated path probabilities and conditional coverages to distinguish trial-wide from path-specific guarantees.
- A trial stops early with probability 0.30. Coverage is 0.80 on the early-stop path and 0.98 on the continuation path. What is unconditional coverage?
- A procedure has 95% unconditional coverage across two paths. Does this imply at least 95% coverage on each path? Explain.
- A selected-treatment path has probability 0.10 and conditional coverage 0.70. The complement path has conditional coverage 0.98. Calculate unconditional coverage.
- Why must the stopping-path probabilities used in a coverage calculation be evaluated at the same parameter value as the conditional coverages?
Solutions. (1) The continuation probability is \(1-0.30=0.70\). Unconditional coverage is \((0.30)(0.80)+(0.70)(0.98)=0.24+0.686=0.926\), or 92.60%. (2) No. The overall value is a weighted average, so one path can have coverage below 95% if another has sufficiently high coverage. (3) The complement probability is 0.90, giving \((0.10)(0.70)+(0.90)(0.98)=0.070+0.882=0.952\), or 95.20%. (4) Both components describe probabilities under the same sampling distribution. Since path probabilities can depend on the true parameter, mixing weights from one parameter value with conditional coverages from another does not give the unconditional coverage at either value.