Confidence Sets by Inverting Asymptotic Tests

From Tests to Confidence Sets

A hypothesis test asks whether a specified parameter value is compatible with the data. A confidence set reverses that question: it collects the parameter values that are not rejected. This construction works for asymptotic tests just as it does for exact tests, and it can produce intervals, disconnected sets, or other shapes.

The key connection is direct. If the true parameter value is included in the inverted set exactly when its null hypothesis is not rejected, then the set’s coverage probability is the test’s probability of accepting that true null. Thus a test with asymptotic size at most $\alpha$ gives a confidence set with asymptotic coverage at least $1-\alpha$. Exact equality requires an additional condition: the rejection probability must converge to $\alpha$, not merely stay below it.

Definition: Let $\Theta$ be the parameter space and let $R_n(\theta_0)$ be the rejection indicator for a test of $H_0:\theta=\theta_0$, based on data of size $n$. The confidence set obtained by inversion is $$ C_n=\{\theta_0\in\Theta:R_n(\theta_0)=0\}. $$ Equivalently, it retains every candidate value whose test does not reject. For a fixed true parameter $\theta$, its coverage probability is $P_\theta(\theta\in C_n)$.

For each fixed candidate value $\theta_0$, a level-$\alpha$ test controls the probability of rejecting when $\theta_0$ is true. In asymptotic work, the corresponding pointwise size condition is

\limsup_{n\to\infty}P_{\theta_0}\{R_n(\theta_0)=1\}\leq\alpha.

This condition is pointwise: it applies at each fixed $\theta_0$. A stronger uniform condition controls the largest rejection probability across the parameter space. The distinction matters when an approximation works well at each fixed interior parameter but poorly near a boundary or as the parameter changes with $n$.

The Coverage–Size Connection

The basic identity follows from the definition, rather than from an additional distributional approximation. It is the reason test inversion works.

Theorem: Suppose $C_n=\{\theta_0\in\Theta:R_n(\theta_0)=0\}$, and the event $\{\theta\in C_n\}$ is measurable for each fixed $\theta$. Then, for every $\theta\in\Theta$, $$ P_\theta(\theta\in C_n)=1-P_\theta\{R_n(\theta)=1\}. $$ Consequently, if $\limsup_{n\to\infty}P_\theta\{R_n(\theta)=1\}\leq\alpha$, then $\liminf_{n\to\infty}P_\theta(\theta\in C_n)\geq1-\alpha$. If the rejection probability converges to $\alpha$, the coverage probability converges to $1-\alpha$.

Proof. By the definition of $C_n$, the event that the true parameter belongs to the set is $\{\theta\in C_n\}=\{R_n(\theta)=0\}$. Since $R_n(\theta)$ is an indicator taking values zero and one, this is the complement of $\{R_n(\theta)=1\}$. Taking probabilities under $P_\theta$ gives the identity. Subtracting the rejection probability from one reverses the limiting inequalities: if its limit superior is at most $\alpha$, the coverage limit inferior is at least $1-\alpha$; if it converges to $\alpha$, coverage converges to $1-\alpha$. $\square$

The same reasoning gives a uniform version. Suppose $\limsup_{n\to\infty}\sup_{\theta\in\Theta}P_\theta\{R_n(\theta)=1\}\leq\alpha$. The identity holds at every $\theta$, so

\inf_{\theta\in\Theta}P_\theta(\theta\in C_n) =1-\sup_{\theta\in\Theta}P_\theta\{R_n(\theta)=1\}.

It follows that the limit inferior of the worst-case coverage is at least $1-\alpha$. Pointwise size control alone does not establish this uniform guarantee: the parameter value at which rejection is most likely may change with $n$.

Worked Inversions

Worked Example: Inverting a Studentized Test for a Mean

Let $X_1,\ldots,X_n$ be independent and identically distributed with mean $\mu$ and finite, positive variance. Write $\bar X$ for the sample mean and $S$ for the sample standard deviation. For each candidate $\mu_0$, use the two-sided test that rejects when $\left|\sqrt n(\bar X-\mu_0)/S\right|>z_{1-\alpha/2}$, where $z_q$ is the $q$-quantile of the standard normal distribution.

Under the null $H_0:\mu=\mu_0$, the central limit theorem and consistency of $S$ give $\sqrt n(\bar X-\mu_0)/S\xrightarrow{d}N(0,1)$. Therefore this is an asymptotic level-$\alpha$ test under the stated assumptions. Inversion retains $\mu_0$ precisely when

\left|\frac{\sqrt n(\bar X-\mu_0)}{S}\right|\leq z_{1-\alpha/2}, \quad\text{or equivalently}\quad \bar X-z_{1-\alpha/2}\frac{S}{\sqrt n} \leq\mu_0\leq \bar X+z_{1-\alpha/2}\frac{S}{\sqrt n}.

For example, take $\bar X=10.4$, $S=2$, $n=25$, and $\alpha=0.05$, so $z_{0.975}\approx1.96$. The estimated standard error is $2/\sqrt{25}=0.4$, and the margin is $1.96(0.4)=0.784$. The inverted set is $[10.4-0.784,10.4+0.784]=[9.616,11.184]$. Its coverage is asymptotically at least 0.95 under the stated conditions; this large-sample argument does not claim exact finite-sample coverage.

Worked Example: Checking Candidate Values in a Binomial Score Test

Suppose $X\sim\operatorname{Bin}(n,p)$, and write $\hat p=X/n$. For an interior candidate $p_0\in(0,1)$, the score test rejects when $\left|(\hat p-p_0)/\sqrt{p_0(1-p_0)/n}\right|>z_{1-\alpha/2}$. For fixed interior $p_0$, the denominator is the null standard deviation of $\hat p$, and the standardized statistic is asymptotically standard normal under $p=p_0$.

Suppose $n=100$, $X=30$, and $\alpha=0.05$. Then $\hat p=0.30$. To check $p_0=0.20$, the standard error under that null is $\sqrt{0.20(0.80)/100}=\sqrt{0.0016}=0.04$. The test statistic is $(0.30-0.20)/0.04=2.5$, which exceeds 1.96, so $0.20$ is rejected and excluded from the inverted set.

For $p_0=0.25$, the null standard error is $\sqrt{0.25(0.75)/100}=\sqrt{0.001875}\approx0.0433$. The statistic is $0.05/0.0433\approx1.155$, which is below 1.96, so $0.25$ is retained. In this example the confidence set consists of all candidates $p_0\in(0,1)$ satisfying $\left|0.30-p_0\right|\leq1.96\sqrt{p_0(1-p_0)/100}$. The coverage guarantee follows from the test’s asymptotic size at the true interior value; the calculations above only determine whether two particular candidates are retained.

Worked Example: An Inverted Test Producing Two Separate Pieces

Let $X_1,\ldots,X_n$ be independent with $X_i\sim N(\theta^2,1)$, where $\theta\in\mathbb R$. For a candidate $\theta_0$, test $H_0:\theta=\theta_0$ using $\left|\sqrt n(\bar X-\theta_0^2)\right|>z_{1-\alpha/2}$. Under this null, $\sqrt n(\bar X-\theta_0^2)\sim N(0,1)$ exactly, so this test has exact size $\alpha$, and also qualifies as an asymptotic test.

With $\bar X=4$, $n=100$, and $\alpha=0.05$, acceptance requires $\left|10(4-\theta_0^2)\right|\leq1.96$. Dividing by 10 gives $\left|4-\theta_0^2\right|\leq0.196$, hence $3.804\leq\theta_0^2\leq4.196$. Because $\theta_0$ may be positive or negative, the inverted set is $[-\sqrt{4.196},-\sqrt{3.804}]\cup[\sqrt{3.804},\sqrt{4.196}]$, approximately $[-2.048,-1.950]\cup[1.950,2.048]$. The set is disconnected even though the test is a conventional two-sided test. Its exact coverage is 0.95 for every $\theta$, because the test’s rejection probability at the true value is exactly 0.05.

Interpreting the Result Carefully

Inverting tests does not guarantee that the result is an interval, bounded, or even nonempty for every possible data set. Those properties depend on how the test statistic varies with the candidate parameter. In the last example the model depends on $\theta^2$, so candidates with opposite signs can be equally compatible with the observations.

Also distinguish a coverage guarantee from a statement about a particular observed set. Before observing the data, $C_n$ is random, and its coverage is the probability that it contains the fixed true parameter. A 95% confidence set does not mean that the realized set has a 95% probability of containing the parameter.

Common pitfall: It is tempting to reason that, because many candidate values are tested, their individual level-$\alpha$ tests require a multiple-testing adjustment before inversion. That is not the coverage argument. For coverage at a fixed true $\theta$, the relevant event is only whether the test of $H_0:\theta=\theta$ rejects: $P_\theta(\theta\in C_n)=1-P_\theta\{R_n(\theta)=1\}$. However, pointwise size control does not automatically imply uniform coverage over $\Theta$, nor does it give simultaneous guarantees for a separate collection of claims.

The practical value of inversion is that it gives a common construction for confidence sets from many test procedures. Wald, score, and likelihood-ratio tests can each be indexed by candidate values and inverted, but their resulting sets need not agree: their rejection regions differ. The size–coverage identity remains the same. What changes is how accurately each test controls rejection probability, especially when asymptotic approximations are used.

Check Your Understanding

Use the definitions and coverage identity to answer the following questions.

For an inverted test, write the event whose probability defines coverage and express it in terms of the rejection event.
If a test’s rejection probability has limit superior at most $0.10$ under each fixed null value, what lower bound follows for the limit inferior of pointwise coverage?
Why does pointwise size control not, by itself, establish a uniform coverage guarantee?
In the binomial example, why is $p_0=0.20$ excluded while $p_0=0.25$ is retained?
What feature of the model $X_i\sim N(\theta^2,1)$ allows the inverted set to have two disconnected pieces?

Confidence Sets by Inverting Asymptotic Tests

What You'll Learn