Deriving the Profile Likelihood-Ratio Confidence Set

From a Likelihood-Ratio Test to a Confidence Set

A likelihood-ratio test asks whether a specified value of a parameter is compatible with the data. A confidence set can be built by asking that question for every possible value and collecting the values that are not rejected. When the parameter of interest is scalar and nuisance parameters are present, the comparison must use the best fit available under each specified target value. That constrained fit is what makes the procedure a profile likelihood-ratio construction.

The key steps are to maximize the likelihood without restriction, maximize it again with the target fixed, compare the two maximized likelihoods, and retain target values whose statistic does not exceed a chi-square cutoff. The resulting set is generally an asymptotic confidence set, not automatically an exact finite-sample one.

Constrained and Unconstrained Maxima

Write the parameter as \(\theta=(\psi,\lambda)\), where \(\psi\) is the scalar target and \(\lambda\) denotes all nuisance parameters. Let \(\Theta\) be the parameter space, and for a proposed target value \(\psi\), define the restricted parameter space \(\Theta_\psi=\{(\psi',\lambda)\in\Theta:\psi'=\psi\}\). The unrestricted maximum log-likelihood is \(\hat\ell=\sup_{\theta\in\Theta}\ell(\theta)\), and the constrained maximum is \(\ell_p(\psi)=\sup_{\theta\in\Theta_\psi}\ell(\theta)\). The latter is the profile log-likelihood. These definitions use suprema so they still make sense if a maximum is not attained; usual likelihood inference requires the relevant maxima to be finite.

Definition: The profile likelihood-ratio statistic for testing the scalar value \(\psi\) is \(W(\psi)=2\{\hat\ell-\ell_p(\psi)\}\). It is nonnegative because the unrestricted parameter space contains the restricted one. Equivalently, if \(L_p(\psi)=\sup_{\theta\in\Theta_\psi}L(\theta)\) and \(\hat L=\sup_{\theta\in\Theta}L(\theta)\), then \(W(\psi)=2\log\{\hat L/L_p(\psi)\}\).

The unrestricted fit is the same for every proposed \(\psi\). The constrained fit changes with \(\psi\), since nuisance parameters are re-estimated subject to that target value. Thus \(W(\psi)\) measures the loss in maximized log-likelihood from imposing the target constraint, rather than the loss from substituting an arbitrary nuisance-parameter value.

Theorem: Suppose the unrestricted and restricted suprema are finite. The likelihood-ratio statistic comparing the unrestricted model with the restriction that the target equals \(\psi\) is exactly \(W(\psi)=2\{\hat\ell-\ell_p(\psi)\}\). If the usual regularity conditions for Wilks' theorem hold, then under a true interior value \(\psi_0\), \(W(\psi_0)\xrightarrow{d}\chi^2_1\).

For the asymptotic assertion, the standard regular setting has a fixed finite-dimensional parameter, an identifiable model, a true parameter in the interior of the parameter space, and a nonsingular Fisher information matrix. The log-likelihood must have the smoothness and local quadratic behavior required for Wilks' theorem; the observations must also satisfy the model's usual sampling and regularity assumptions. The restriction \(\psi=\psi_0\) must be a regular restriction of codimension one. Boundary values, non-identifiability, or singular information can invalidate the stated chi-square approximation. The result is asymptotic; it does not claim that \(W(\psi_0)\) has a chi-square distribution in every finite sample.

Inverting the Cutoff

Let \(0<\alpha<1\), and let \(c_\alpha\) be the \(1-\alpha\) quantile of a chi-square distribution with one degree of freedom. For a regular scalar target, \(c_\alpha=\chi^2_{1,1-\alpha}\). Inverting the likelihood-ratio test means retaining every proposed target value whose statistic is at or below this cutoff.

Definition: The asymptotic profile likelihood-ratio confidence set with nominal confidence level \(1-\alpha\) is \(C_{1-\alpha}=\{\psi\in\Psi:W(\psi)\leq c_\alpha\}\), where \(\Psi\) is the set of allowed target values. It need not be a single bounded interval.

The next derivation makes the test-inversion logic explicit. A value \(\psi\) is excluded exactly when its likelihood-ratio statistic exceeds the test cutoff. The set of values not excluded is therefore the confidence set.

Fix a candidate target.
Test \(H_0:\psi=\psi_0\) by comparing the best fit in \(\Theta_{\psi_0}\) with the best fit in \(\Theta\). The statistic is \(W(\psi_0)\).

Apply the asymptotic cutoff.
Under the null and the stated regularity conditions, reject at approximate level \(\alpha\) when \(W(\psi_0)>c_\alpha\).

Invert all these tests.
Keep each \(\psi_0\) for which \(W(\psi_0)\leq c_\alpha\). The retained values form \(C_{1-\alpha}\).

Under the same regularity conditions, the coverage statement follows directly. If the true target is \(\psi_0\), then \(P_{\psi_0}(\psi_0\in C_{1-\alpha})=P_{\psi_0}\{W(\psi_0)\leq c_\alpha\}\). By Wilks' theorem, \(W(\psi_0)\xrightarrow{d}\chi^2_1\). The chi-square distribution has a continuous distribution function at \(c_\alpha\), so the probability on the right converges to \(P(\chi^2_1\leq c_\alpha)=1-\alpha\). Hence the coverage tends to the nominal level as sample size increases; this argument does not establish exact finite-sample coverage. \(\square\)

Worked Constructions

Worked Example: A Poisson Rate With Exposure

Suppose \(Y\sim\operatorname{Poisson}(10\lambda)\), where \(\lambda>0\) is the event rate per unit exposure. The observed count is \(Y=40\), and the target is \(\lambda\). Ignoring terms that do not depend on \(\lambda\), the log-likelihood is \(\ell(\lambda)=40\log\lambda-10\lambda\).

Differentiate: \(\ell'(\lambda)=40/\lambda-10\), so the unrestricted maximum occurs at \(\hat\lambda=4\). There are no nuisance parameters here, so the constrained maximum at a proposed \(\lambda\) is simply \(\ell_p(\lambda)=\ell(\lambda)\). The statistic is \(W(\lambda)=2\{\ell(4)-\ell(\lambda)\}=2\{40\log(4/\lambda)-40+10\lambda\}\).

At the 95% cutoff \(c_{0.05}\approx3.841\), the endpoints solve \(W(\lambda)=3.841\). Put \(m=10\lambda\), so the equation becomes \(2\{40\log(40/m)-40+m\}=3.841\). Numerical solutions are approximately \(m=28.8507\) and \(m=53.705\). Substitution gives statistics approximately \(3.841\) at each root. Dividing by the exposure 10 gives the approximate profile confidence set \([2.885,5.371]\) for \(\lambda\), to three decimal places. Values between the roots satisfy the cutoff; values outside do not.

Worked Example: A Binomial Likelihood-Ratio Check

Let \(Y\sim\operatorname{Binomial}(20,p)\), and suppose \(Y=14\). The unrestricted estimate is \(\hat p=14/20=0.7\). For a proposed \(p\), the log-likelihood, up to a constant, is \(14\log p+6\log(1-p)\), giving \(W(p)=2\{14\log(0.7/p)+6\log(0.3/(1-p))\}\).

To check whether \(p=0.5\) is in the 95% profile likelihood set, calculate \(W(0.5)=2\{14\log(1.4)+6\log(0.6)\}\). Using \(\log(1.4)\approx0.336472\) and \(\log(0.6)\approx-0.510826\), the quantity in braces is approximately \(14(0.336472)+6(-0.510826)=4.710608-3.064956=1.645652\). Thus \(W(0.5)\approx3.2913\), which is below \(3.841\); this target value is retained by the asymptotic 95% likelihood-ratio rule. This calculation checks membership, not the exact endpoints, and the chi-square calibration remains an approximation for this sample size.

Worked Example: Profiling an Intercept When the Slope Is the Target

Consider four independent normal responses with known variance 1, modeled by \(Y_i\sim N(\beta_0+\beta_1x_i,1)\). The covariates are \((-1,-1,1,1)\) and the observations are \((1,2,3,5)\). Take the slope \(\psi=\beta_1\) as the target and the intercept \(\beta_0\) as nuisance. The unrestricted least-squares estimates are \(\hat\beta_0=\bar y=2.75\) and \(\hat\beta_1=\sum_i x_i(y_i-\bar y)/\sum_i x_i^2=5/4=1.25\).

At a fixed slope \(b\), the best intercept is the mean of \(y_i-bx_i\). Since \(\sum_i x_i=0\), this is \(2.75\) for every \(b\). The unrestricted fitted values are \(1.5,1.5,4,4\), giving residual sum of squares \(0.25+0.25+1+1=2.5\). At slope \(b\), the residuals differ from those at slope \(1.25\) by \(-(b-1.25)x_i\). The cross-product of the unrestricted residuals with \(x\) is zero, so the squared residual sum is \(\operatorname{SSE}_p(b)=2.5+(b-1.25)^2\sum_i x_i^2=2.5+4(b-1.25)^2\).

For known variance 1, the log-likelihood difference gives \(W(b)=\operatorname{SSE}_p(b)-2.5=4(b-1.25)^2\). Applying the 95% cutoff, \(4(b-1.25)^2\leq3.841\), or \(|b-1.25|\leq\sqrt{3.841/4}\approx0.980\). The resulting set is approximately \([0.270,2.230]\). In this normal linear model the likelihood-ratio calculation can also be checked directly from the displayed quadratic expression; the example illustrates how re-estimating the intercept enters the profile.

What the Set Says, and What It Does Not

A profile likelihood confidence set is built from relative likelihood, not from a fixed standard error around the estimate. It respects the parameter space and compares each target value with the best possible fit overall. The set may be asymmetric because the log-likelihood need not be quadratic over the relevant range; it may also fail to be a single bounded interval. Those shapes do not change the defining rule \(W(\psi)\leq c_\alpha\).

Its nominal coverage is justified by the reference distribution of the likelihood-ratio statistic under the true target. In finite samples, especially when information is weak or regularity is doubtful, the cutoff may give imperfect coverage. The chi-square rule should therefore be reported as an asymptotic calibration rather than described as an exact guarantee.

Common Mistakes

Holding nuisance parameters at their unrestricted estimates. This does not generally calculate \(\ell_p(\psi)\). For each fixed target value, maximize over the nuisance parameters subject to that constraint.
Integrating rather than maximizing. A profile likelihood uses a constrained supremum. Integrating nuisance parameters produces a different object.
Reversing the likelihood ratio. The statistic is twice the unrestricted log-likelihood minus the constrained log-likelihood, so it is nonnegative. A value is retained when the statistic is less than or equal to the cutoff.
Calling the chi-square interval exact without justification. Wilks' theorem supplies an asymptotic reference distribution under regularity conditions; it does not by itself prove finite-sample coverage.
Using the wrong degrees of freedom. The scalar target imposes one regular restriction, so the usual cutoff is \(\chi^2_1\), even when the nuisance parameter vector has several components.

Key takeaway: For each proposed scalar target value, compare the unrestricted maximum likelihood with the maximum under that value. Retain exactly those values whose twice-log-likelihood loss is no greater than the appropriate chi-square cutoff.

Check Your Understanding

For each question, identify the relevant likelihood comparison and state any calibration qualification that applies.

Write the profile likelihood-ratio statistic when \(\theta=(\psi,\lambda)\), with \(\psi\) scalar.
Why must nuisance parameters be re-estimated when calculating the constrained maximum?
How does test inversion produce the set of target values retained at level \(1-\alpha\)?
What regularity issue could make the usual chi-square-one cutoff unreliable?
In the binomial example, explain why \(p=0.5\) is retained under the displayed cutoff.
In the Gaussian regression example, identify which parameter is constrained and which one is profiled out.