Likelihood-Based Confidence Regions by Test Inversion

From Tests to Confidence Regions

A test asks whether a specified parameter value is compatible with the data at a chosen significance level. A confidence region reverses that question: it collects the parameter values that would not be rejected by the test. This gives a practical route from Wald, likelihood-ratio, and score tests to uncertainty statements about one or several parameters.

The distinction between a joint region and a collection of separate intervals matters. A joint region for a vector parameter is designed to cover the entire true vector together. Separate intervals target each coordinate individually, and their individual coverage levels do not generally add up to the same simultaneous coverage.

Definition: Let $\theta$ be a parameter vector and let $T_n(\theta_0)$ be a test statistic for $H_0:\theta=\theta_0$. At significance level $\alpha$, the confidence set obtained by test inversion is the set of candidate values not rejected by the test: $$ C_{1-\alpha}=\{\theta_0:T_n(\theta_0)\leq c_{1-\alpha}\}, $$ where $c_{1-\alpha}$ is the test's critical value. The set is a confidence region when $\theta$ has multiple components.

In the regular large-sample settings considered here, the critical value for a test of $q$ restrictions is the $1-\alpha$ quantile of a chi-square distribution with $q$ degrees of freedom. Write this as $\chi^2_{q,1-\alpha}$. Its use requires the relevant likelihood regularity conditions, an interior true parameter, identification, and nonsingular information for the restrictions being tested. The large-sample coverage is approximate, not an exact finite-sample guarantee.

Three Inversion Rules

For a $k$-dimensional parameter vector, let $\hat{\theta}$ be the maximum likelihood estimator (MLE), $\ell_n(\theta)$ the log likelihood, and $\widehat V$ an estimated covariance matrix for $\hat{\theta}$. Under regularity conditions, the three tests of a candidate value $\theta_0$ give the following full-vector statistics.

Test-inversion formulas:

Wald: $W_n(\theta_0)=(\hat{\theta}-\theta_0)'\widehat V^{-1}(\hat{\theta}-\theta_0)$.
Likelihood ratio: $LR_n(\theta_0)=2\{\ell_n(\hat{\theta})-\ell_n(\theta_0)\}$.
Score: $S_n(\theta_0)=s_n(\theta_0)'I_n(\theta_0)^{-1}s_n(\theta_0)$, where $s_n(\theta_0)$ is the score vector and $I_n(\theta_0)$ is the Fisher information matrix.

For each full-vector test, the asymptotic reference distribution is $\chi^2_k$. The corresponding confidence region contains the candidates whose statistic does not exceed $\chi^2_{k,1-\alpha}$.

The likelihood-ratio statistic compares the maximized log likelihood with the log likelihood under the candidate value. The score statistic instead evaluates the score and information at that candidate, without requiring an unrestricted estimate under the alternative for each test. If the hypothesis imposes $q$ smooth restrictions rather than specifying the entire vector, the test has $q$ degrees of freedom under the same regularity framework. In score tests with nuisance parameters, the nuisance component must be accounted for using the appropriate efficient score and information; using an unadjusted score can give the wrong test.

The three resulting regions need not have the same shape or boundaries in a finite sample. Under standard regularity conditions they are often asymptotically equivalent to first order near the true parameter, but this does not make them interchangeable in every sample or parameterization.

Worked Example: Inverting a Wald Test to Obtain an Ellipse

Suppose a regular two-parameter model yields the hypothetical estimate $\hat{\theta}=(2.00,-1.00)'$, with covariance estimate $$ \widehat V= \begin{pmatrix} 0.0400 & 0\\ 0 & 0.0100 \end{pmatrix}. $$ The first parameter is measured in hours and the second in dollars per unit. Construct the 95% Wald region.

Step 1: Check the ingredients. The covariance estimate is positive definite, since its diagonal entries are positive. Its inverse is $$ \widehat V^{-1}= \begin{pmatrix} 25 & 0\\ 0 & 100 \end{pmatrix}. $$ There are two parameters, so the asymptotic 95% cutoff is $\chi^2_{2,0.95}=5.991$.

Step 2: Invert the test. For a candidate $(\theta_1,\theta_2)'$, the Wald statistic is $$ W_n(\theta)=25(\theta_1-2.00)^2+100(\theta_2+1.00)^2. $$ Thus the 95% joint Wald region is $$ 25(\theta_1-2.00)^2+100(\theta_2+1.00)^2\leq 5.991. $$ This is an ellipse centered at the estimate. Its coordinate half-widths, obtained by setting the other coordinate's deviation to zero, are $\sqrt{5.991(0.0400)}=0.4895$ hours and $\sqrt{5.991(0.0100)}=0.2448$ dollars per unit.

For comparison, separate two-sided 95% Wald intervals use the normal critical value 1.96. Their half-widths are $1.96\sqrt{0.0400}=0.3920$ hours and $1.96\sqrt{0.0100}=0.1960$ dollars per unit. These are narrower marginal intervals, not the coordinate projections of the joint 95% ellipse.

Why Inversion Gives Coverage

The coverage argument follows directly from the definition of the set. Suppose the true parameter is $\theta$, and form the confidence set by inverting a level-$\alpha$ test at every candidate value. The event that the true value belongs to the set is exactly the event that the test does not reject the null hypothesis when the candidate equals the truth.

Evaluate the test at the truth.
When the candidate is $\theta$, the tested null is $H_0:\theta=\theta$, which is true.

Use the test's size.
A level-$\alpha$ test rejects a true null with probability approximately $\alpha$ in the large sample under the stated conditions.

Translate non-rejection into coverage.
Because $\theta\in C_{1-\alpha}$ exactly when the test at $\theta$ does not reject, $P_\theta(\theta\in C_{1-\alpha})$ is approximately $1-\alpha$.

This argument also explains the interpretation: in repeated samples from the model, the procedure produces regions containing the true parameter about $1-\alpha$ of the time. It does not assign a probability of $1-\alpha$ to the fixed parameter being inside the particular region already observed.

Worked Example: Reading an LR Confidence Contour

Consider a hypothetical regular two-parameter likelihood with log likelihood $$ \ell_n(\theta_1,\theta_2)=C-(\theta_1-1)^2-(\theta_2-2)^2-0.2(\theta_1-1)^4, $$ where $C$ does not depend on the parameters. The terms subtracted from $C$ are nonnegative and equal zero at $(1,2)$, so the unrestricted maximum occurs at $\hat{\theta}=(1,2)$. Find the LR-region membership of two candidates at the asymptotic 95% level.

The LR statistic is twice the difference between the maximum log likelihood and the candidate log likelihood: $$ LR_n(\theta_1,\theta_2)=2\left\{(\theta_1-1)^2+(\theta_2-2)^2+0.2(\theta_1-1)^4\right\}. $$ With two restrictions, the cutoff is 5.991.

At $(2,2)$, the statistic is $$ LR_n(2,2)=2\{1^2+0^2+0.2(1^4)\}=2.400. $$ Since $2.400<5.991$, this candidate is in the inverted LR region.

At $(2.5,2)$, it is $$ LR_n(2.5,2)=2\{1.5^2+0^2+0.2(1.5^4)\} =2\{2.25+1.0125\}=6.525. $$ Since $6.525>5.991$, this candidate is excluded. The confidence region is the full set of candidates satisfying the LR inequality; checking two points illustrates membership but does not replace describing that set.

Score-Statistic Inversion

Inverting the score test requires evaluating both the score and the information at each candidate null value. The following binomial example makes those calculations explicit. Its chi-square calibration is an asymptotic one; it is not an exact small-sample confidence procedure.

Worked Example: Inverting a Two-Parameter Score Test

Suppose two independent samples have $X_1\sim\operatorname{Binomial}(100,p_1)$ and $X_2\sim\operatorname{Binomial}(100,p_2)$, with observed counts $x_1=40$ and $x_2=30$. Assume both probabilities are interior, so candidate values satisfy $0<p_1,p_2<1$. For one binomial sample, the score at candidate $p$ is $(x-np)/\{p(1-p)\}$, and the Fisher information is $n/\{p(1-p)\}$. Independence makes the two-parameter information matrix diagonal.

For candidate $(p_1,p_2)$, substitution into the score quadratic gives $$ S_n(p_1,p_2)= \frac{(40-100p_1)^2}{100p_1(1-p_1)} + \frac{(30-100p_2)^2}{100p_2(1-p_2)}. $$ The 95% score confidence region contains candidates for which this statistic is at most $\chi^2_{2,0.95}=5.991$.

At $(0.50,0.40)$, the first contribution is $10^2/(100\cdot0.50\cdot0.50)=4.000$, and the second is $10^2/(100\cdot0.40\cdot0.60)=4.167$. Therefore $S_n=8.167$, which exceeds 5.991; this candidate is excluded.

At $(0.45,0.35)$, each count differs from its candidate expected count by 5. The contributions are $25/(100\cdot0.45\cdot0.55)=1.010$ and $25/(100\cdot0.35\cdot0.65)=1.099$, giving $S_n=2.109$. This candidate is included because $2.109<5.991$. The inverted region is defined by the displayed inequality over the interior probability space.

Joint Regions and Marginal Intervals

A joint region answers whether a complete vector of parameter values is plausible together. For example, the Wald ellipse in the first worked example targets simultaneous coverage of both coordinates. Its shape reflects their estimated uncertainty and covariance; with nonzero covariance, the ellipse is tilted rather than aligned with the coordinate axes.

Separate 95% intervals answer a different question for each coordinate. Each interval has approximately 95% marginal coverage under its own procedure, but the probability that both cover simultaneously is generally below 95%. If two interval-coverage events were independent, the probability that both cover would be $0.95^2=0.9025$; in practice their dependence determines the joint probability. Marginal coverage alone does not establish simultaneous coverage.

A projection of a joint $1-\alpha$ region onto one coordinate has coverage at least $1-\alpha$: whenever the full true vector is in the region, its true coordinate must be in that projection. The projection can be wider than a separately constructed marginal interval. Conversely, an individual marginal interval is not automatically a projection of a joint region with the same nominal level.

These distinctions matter when conclusions concern several coefficients at once, such as whether a group of parameters jointly equals specified values. Use a joint test and its inverted region for a simultaneous claim. Use marginal intervals when the inferential target is a single coordinate at a time, and do not describe a collection of them as a joint 95% statement without a procedure that provides that coverage.

Common Mistake: Treating Separate Intervals as a Joint Region

A common error is to compute a 95% interval for each of two parameters and then state that the pair is covered with 95% confidence. That conclusion does not follow: each interval has its own marginal coverage, while the probability that both cover together depends on their joint behavior. Use a joint test inversion, or an appropriate simultaneous procedure, when the claim concerns the whole parameter vector.

Key takeaway: Wald, LR, and score confidence regions are sets of candidate parameter values that pass their corresponding tests. Their coverage is a repeated-sampling property, and a joint region's simultaneous coverage is not the same as the marginal coverage of separate intervals.

Check Your Understanding

Use the definitions and calculations above to answer the following.

For a three-parameter full-vector test at level 5%, which chi-square degrees of freedom determine the asymptotic cutoff?
In a Wald region, what role does the estimated covariance matrix play in the region's shape?
For the LR likelihood in the worked example, would the candidate $(2,2)$ be included at the 95% level? Explain using its statistic and cutoff.
Why does a candidate score test evaluate the score and information at the candidate parameter value?
Why do two separate 95% intervals not generally provide 95% simultaneous coverage for both parameters?