Exact Confidence Sets by Test Inversion

From Tests to Confidence Sets

An exact conditional test answers whether a particular parameter value is compatible with the observed data, at a specified error level. Repeating that test for every possible parameter value turns the collection of values not rejected into a confidence set. This construction is called test inversion. Its coverage guarantee follows directly from the size guarantee of the tests; discreteness determines how often the resulting coverage is greater than, rather than exactly equal to, its nominal level.

Let $S$ denote the observed statistic and $T$ the statistic being conditioned on. For a fixed observed value $t$, write the conditional distribution as $f_\theta(s\mid T=t)$, with support $\mathcal S_t$. The conditioning is important: the test and its size are evaluated under this conditional distribution, not under the original unconditioned model. For each candidate value $\theta_0$, suppose a test specifies a rejection region $R_{\theta_0,t}\subseteq\mathcal S_t$.

Definition: The confidence set obtained by inverting these tests is $$ C(s,t)=\{\theta_0: s\notin R_{\theta_0,t}\}. $$ It contains exactly the parameter values whose tests do not reject at the observed statistic. If, for every $\theta_0$ and every relevant $t$, the test has conditional size at most $\alpha$, then the inverted set has conditional coverage at least $1-\alpha$.

A confidence set need not be an interval in every model. It becomes an interval when the tests and conditional distributions have suitable ordering and monotonicity properties. We will make those properties explicit in the examples rather than assuming that inversion always produces two numerical endpoints.

Why Inversion Guarantees Coverage

The essential argument is short, but it relies on matching the event “the true value is absent from the set” to the rejection event for the test of that value. The guarantee is conditional and holds separately for each value of the conditioning statistic for which the conditional model is defined.

Theorem: Suppose that for every $\theta_0$ and $t$, $$ P_{\theta_0}(S\in R_{\theta_0,t}\mid T=t)\leq\alpha. $$ Then the set $C(S,T)=\{\theta_0:S\notin R_{\theta_0,T}\}$ satisfies $$ P_{\theta_0}\{\theta_0\in C(S,T)\mid T=t\}\geq 1-\alpha $$ for every $\theta_0$ and every relevant $t$. Thus its conditional coverage is at least $1-\alpha$.

Proof. Fix $\theta_0$ and $t$. By the definition of the inverted set, the event that the true value is not in the set is precisely the event that the test of that value rejects: $$ \{\theta_0\notin C(S,t)\}=\{S\in R_{\theta_0,t}\}. $$ Taking conditional probabilities and using the test-size assumption gives $$ P_{\theta_0}\{\theta_0\notin C(S,T)\mid T=t\} =P_{\theta_0}(S\in R_{\theta_0,t}\mid T=t)\leq\alpha. $$ Taking complements proves the claimed lower bound on conditional coverage. $\square$

This argument does not require a continuous distribution, a likelihood approximation, or a large sample. It does require that the rejection rule really have size at most $\alpha$ under the stated conditional law. If the rule is only approximately calibrated, the resulting coverage guarantee is likewise only approximate.

Worked Example: A One-Sided Exact Upper Limit

Worked Example: Inverting the Correct Tail for an Upper Limit

Suppose $X\mid N=5\sim\operatorname{Binomial}(5,p)$, and the observed count is $x=0$. Construct a 95% one-sided upper confidence limit for $p$ by inverting exact tests.

To obtain an upper limit, test each candidate $p_0$ against smaller values of $p$. Small counts are evidence for the alternative $p<p_0$, so the test rejects for small $X$. Use the exact lower-tail probability $P_{p_0}(X\leq x)$ and reject when it is strictly less than $0.05$. This strict convention includes a parameter value at equality in the inverted set; the test still has size at most $0.05$.

For the observed value $x=0$, $$ P_{p_0}(X\leq 0)=P_{p_0}(X=0)=(1-p_0)^5. $$ The values not rejected therefore satisfy $$ (1-p_0)^5\geq 0.05. $$ Since $1-p_0$ is nonnegative for $0\leq p_0\leq1$, taking fifth roots and rearranging gives $$ p_0\leq 1-0.05^{1/5}\approx 1-0.54928=0.45072. $$ The inverted set is $[0,0.45072]$, a one-sided 95% exact upper confidence set under this test convention.

The direction of the test matters. A test against larger $p$ would reject for large $X$; observing $x=0$ would not be rejected for any candidate $p_0$, and that test would not produce the upper limit just calculated. The upper limit above comes from the lower-tail test against smaller $p$.

Equal-Tailed Confidence Limits

For a scalar parameter and an ordered statistic, a common two-sided construction allocates $\alpha/2$ to each tail. In the binomial example, $X\mid N=n\sim\operatorname{Binomial}(n,p)$ is stochastically increasing in $p$: larger values of $p$ tend to produce larger counts. For observed $x$, a small count is evidence against a candidate $p$ in the direction of smaller values, while a large count is evidence in the direction of larger values.

One nonrandomized equal-tailed rule rejects a candidate $p_0$ if either $P_{p_0}(X\leq x)<\alpha/2$ or $P_{p_0}(X\geq x)<\alpha/2$. Each tail rejection event has probability at most $\alpha/2$ under $p_0$: the probability of a lower-tail event whose tail probability is below $\alpha/2$ cannot exceed $\alpha/2$, and the same reasoning applies to the upper tail. By the union bound, the probability of rejection from either tail is at most $\alpha$. Inverting this test yields the set of $p_0$ for which both tail probabilities are at least $\alpha/2$. Including equality is consistent with the strict rejection rule above.

Result: If each of two one-sided tests has conditional size at most $\alpha/2$, their union has conditional size at most $\alpha$. The confidence set obtained by inverting the union therefore has conditional coverage at least $1-\alpha$. In discrete models, the tail probabilities generally jump in steps, so this coverage need not equal $1-\alpha$.

Worked Example: Equal-Tailed Limits for Four Binomial Trials

Let $X\sim\operatorname{Binomial}(4,p)$, observe $x=1$, and construct a 90% equal-tailed confidence set. Here $\alpha=0.10$, so each tail uses $0.05$.

The lower endpoint is determined by the upper-tail test: a candidate $p$ is rejected in the direction of larger values when $P_p(X\geq1)<0.05$. Since $$ P_p(X\geq1)=1-P_p(X=0)=1-(1-p)^4, $$ the lower endpoint solves $1-(1-p)^4=0.05$. Thus $$ p_L=1-0.95^{1/4}\approx 0.01274. $$

The upper endpoint is determined by the lower-tail test, which rejects in the direction of smaller values when $P_p(X\leq1)<0.05$. Here $$ P_p(X\leq1)=(1-p)^4+4p(1-p)^3. $$ Solving $(1-p)^4+4p(1-p)^3=0.05$ gives $p_U\approx0.75137$. For example, at $p=0.75137$, the left-hand side is approximately $0.05000$, confirming the root to the displayed precision. The inverted equal-tailed set is approximately $[0.01274,0.75137]$.

Each endpoint comes from a different directional test: the lower endpoint from the upper tail, and the upper endpoint from the lower tail. Reversing either direction would invert the wrong test and would not give the stated endpoint.

Discreteness and Conservative Coverage

For a continuous statistic, a tail probability can often vary smoothly enough that a test reaches its nominal size at a boundary. In a discrete distribution, tail probabilities change only when the observed statistic crosses a possible value. A nonrandomized rejection region may therefore have probability strictly less than its size bound. Inverting such tests still guarantees at least the nominal coverage, but can produce coverage above it. “Exact” means the finite-sample error bound is valid; it does not mean the coverage is exactly $1-\alpha$ at every parameter value.

Worked Example: Coverage Above 90% in a Discrete Model

Let $X\sim\operatorname{Binomial}(2,p)$. For each observed $x$, construct a one-sided 90% upper confidence limit by inverting the lower-tail test against smaller $p$, using the rule to reject $p_0$ when $P_{p_0}(X\leq x)<0.10$.

For $x=0$, the endpoint $U(0)$ solves $$ P_{U(0)}(X\leq0)=(1-U(0))^2=0.10, $$ so $U(0)=1-\sqrt{0.10}\approx0.68377$. For $x=1$, $$ P_p(X\leq1)=1-p^2, $$ so $1-U(1)^2=0.10$, giving $U(1)=\sqrt{0.90}\approx0.94868$. For $x=2$, $P_p(X\leq2)=1$ for every $p$, so no candidate is rejected and $U(2)=1$.

At the true value $p=0.5$, every possible observation produces an upper limit at least $0.5$: $U(0)\approx0.68377$, $U(1)\approx0.94868$, and $U(2)=1$. The confidence set therefore covers $p=0.5$ with probability $1$, not merely $0.90$. This is overcoverage caused by the discrete support and the nonrandomized rejection rule.

At $p=0.8$, the limit fails to cover only when $X=0$, because $U(0)\approx0.68377<0.8$, whereas $U(1)$ and $U(2)$ exceed $0.8$. Hence the coverage is $$ P_{0.8}(X\geq1)=1-P_{0.8}(X=0)=1-(0.2)^2=0.96. $$ This too exceeds $0.90$, while illustrating that the amount of conservatism can depend on the true parameter.

Using Exact Confidence Sets Carefully

In a conditional model, the coverage statement is conditional on the value of the statistic used for conditioning. If that conditional distribution removes nuisance parameters, the resulting guarantee is correspondingly free of those nuisance parameters within the stated conditional model. The guarantee does not, by itself, imply that the set is narrow, that its endpoints are unique, or that it is an interval. Those properties depend on the conditional family and the tests being inverted.

A common mistake is to select an endpoint equation without checking which alternative and rejection tail produced it. For an upper confidence limit based on a lower-tail test against smaller parameter values, the upper endpoint is where the lower-tail probability reaches its cutoff. A large-tail test against larger values instead addresses the opposite direction. Always identify the test first, then invert its non-rejection condition.

Equal-tailed limits are also not the only way to allocate error between two tails. Their appeal is transparent allocation, not guaranteed shortest length or exact nominal coverage. Since exact nonrandomized tail probabilities are discrete, the resulting confidence set is often conservative. This is a finite-sample property of the construction, not evidence that the test or inversion has failed.

Check Your Understanding

Use the test direction, conditional size, and inversion rule to answer these questions.

For a one-sided upper confidence limit in a binomial model, which alternative and rejection tail should be used to obtain an upper endpoint by inversion?
State the event identity that makes the coverage proof for an inverted test work.
Why can a nonrandomized exact confidence set have coverage greater than its nominal level?
For $X\sim\operatorname{Binomial}(3,p)$ and observed $x=0$, write the equation for a one-sided 95% upper limit formed by inverting a lower-tail test against smaller $p$.
Does a conditional coverage guarantee necessarily imply that the confidence set is an interval? Explain.