Profile-Likelihood Confidence Intervals for Scalar Targets

Why Profile a Likelihood?

A model may have several unknown parameters, even when the scientific question concerns only one scalar quantity. For example, a rate ratio may be the target while a baseline rate is unknown. An interval for the target should reflect the fact that the baseline rate was estimated too. Profile likelihood does this by optimizing over nuisance parameters separately at every candidate value of the target.

The resulting interval is obtained by inverting a likelihood-ratio test: retain the candidate target values whose likelihood is not too far below the unrestricted maximum. Unlike a symmetric interval formed by adding and subtracting a standard error, a profile-likelihood interval can have different distances from its estimate to its two endpoints.

Definition: Let $\theta$ be a parameter vector, let $\psi=g(\theta)$ be a scalar target, and let $\ell(\theta)$ be the log likelihood. The profile log likelihood at a candidate value $\psi_0$ is $$ \ell_p(\psi_0)=\sup_{\theta:\,g(\theta)=\psi_0}\ell(\theta). $$ The unrestricted maximum is $\ell(\hat{\theta})=\sup_\theta \ell(\theta)$, and the likelihood-ratio statistic for testing $H_0:g(\theta)=\psi_0$ is $$ LR(\psi_0)=2\{\ell(\hat{\theta})-\ell_p(\psi_0)\}. $$

When the target is one component of $\theta=(\psi,\lambda)$, the nuisance parameter or parameters are denoted by $\lambda$, and profiling means maximizing $\ell(\psi,\lambda)$ over $\lambda$ with $\psi$ held fixed. If the target is a function $g(\theta)$, the maximization is over all parameter values satisfying $g(\theta)=\psi_0$. In either case, the profile is built from the same likelihood as the unrestricted fit.

Inverting the Likelihood-Ratio Test

Under the usual regularity conditions for likelihood-based inference, the likelihood-ratio statistic for a scalar restriction is approximately distributed as a chi-square random variable with one degree of freedom when the null hypothesis is true. These conditions include an identifiable, sufficiently smooth model and a true parameter in the interior of the parameter space. For a nominal $100(1-\alpha)\%$ interval, write $\chi^2_{1,1-\alpha}$ for the $1-\alpha$ quantile of that distribution.

Theorem: Inverting the asymptotic likelihood-ratio test gives the confidence set $$ \mathcal{C}_{1-\alpha} = \left\{\psi_0: 2\{\ell(\hat{\theta})-\ell_p(\psi_0)\} \leq \chi^2_{1,1-\alpha} \right\}. $$ For a 95% interval, $\alpha=0.05$ and $\chi^2_{1,0.95}\approx 3.841$. If the retained set is a single connected interval, its endpoints are the values where the likelihood-ratio statistic equals the cutoff.

The degrees of freedom are one because the null hypothesis fixes one scalar target, not because the model has one parameter. The nuisance parameters are re-estimated under each fixed target value; they are not counted as additional restrictions. If the retained set is disconnected, or reaches a parameter-space boundary, it need not be a single ordinary interval, and the model’s assumptions and the shape of the set require attention.

The inversion argument is direct. For each fixed $\psi_0$, the test rejects when $LR(\psi_0)>\chi^2_{1,1-\alpha}$. Therefore, the set of values not rejected is exactly the set displayed above. Under the stated regularity conditions, the test has approximate rejection probability $\alpha$ when $\psi_0$ is the true target. Consequently, the inverted set has approximate coverage $1-\alpha$. This is a repeated-sampling statement about the procedure, not a probability that a fixed parameter lies in the interval after the data have been observed.

Find the unrestricted fit.
Maximize the log likelihood over all model parameters to obtain $\hat{\theta}$ and $\ell(\hat{\theta})$.

Profile at a candidate target value.
For each $\psi_0$, maximize the log likelihood subject to $g(\theta)=\psi_0$, giving $\ell_p(\psi_0)$.

Compare with the cutoff.
Retain $\psi_0$ if $2\{\ell(\hat{\theta})-\ell_p(\psi_0)\}\leq\chi^2_{1,1-\alpha}$.

Locate the endpoints.
For a single interval, solve the equality at the lower and upper crossings, typically by numerical optimization and root-finding.

Worked Examples

Worked Example: A Poisson Rate Ratio with an Unknown Baseline Rate

Suppose two independent hypothetical counts satisfy $X\sim\operatorname{Pois}(\lambda\psi)$ and $Y\sim\operatorname{Pois}(\lambda)$, where $\lambda>0$ is an unknown baseline rate and $\psi>0$ is the rate ratio of interest. Assume the counts arise over equal exposure periods, the observations are independent, and the Poisson model is appropriate. The hypothetical observed counts are $x=18$ and $y=12$. Construct an approximate 95% profile-likelihood interval for $\psi$.

Ignoring terms that do not depend on the parameters, the log likelihood is

\ell(\psi,\lambda) =18\log(\lambda\psi)-\lambda\psi +12\log(\lambda)-\lambda.

The unrestricted maximum is $\hat{\psi}=18/12=1.5$ and $\hat{\lambda}=12$, so the fitted means are 18 and 12. At a fixed candidate $\psi$, differentiating with respect to $\lambda$ gives $30/\lambda-(\psi+1)=0$, hence $\hat{\lambda}(\psi)=30/(\psi+1)$. The constrained fitted means are therefore $30\psi/(\psi+1)$ and $30/(\psi+1)$.

Substituting those fitted means into the likelihood-ratio statistic gives

LR(\psi) = 2\left[ 18\log\left(\frac{18(\psi+1)}{30\psi}\right) + 12\log\left(\frac{12(\psi+1)}{30}\right) \right].

For a 95% interval, retain $\psi$ when this quantity is at most 3.841. Numerically solving $LR(\psi)=3.841$ on either side of $\hat{\psi}=1.5$ gives endpoints approximately $0.730$ and $3.20$. For example, at the lower endpoint the constrained baseline estimate is $30/(1+0.730)\approx17.34$; at the upper endpoint it is $30/(1+3.20)\approx7.14$. Both are obtained by re-estimating the nuisance parameter, not by holding it at its unrestricted estimate of 12.

Thus the approximate 95% profile-likelihood interval for the rate ratio is $[0.730,\,3.20]$. It is notably asymmetric around 1.5. The interval includes 1, so these data do not exclude equal rates at the corresponding approximate 5% two-sided likelihood-ratio level. The interval is approximate because the chi-square calibration relies on large-sample regularity; the counts here are hypothetical and do not establish that the approximation is accurate for every application.

Worked Example: Profiling a Correlated Nuisance Mean

Consider $n=10$ hypothetical independent observations from a bivariate normal distribution with mean $(\psi,\lambda)$ and known covariance matrix

\Sigma= \begin{pmatrix} 4 & 1.2\\ 1.2 & 1 \end{pmatrix}.

The covariance matrix is positive definite because its determinant is $4-1.2^2=2.56>0$. Suppose the observed sample mean is $(\bar{x}_1,\bar{x}_2)=(4.20,1.10)$. The target is the first mean $\psi$; the second mean $\lambda$ is nuisance. The model assumptions are independent bivariate normal observations and a known covariance matrix.

For a fixed $\psi$, maximizing the normal likelihood over $\lambda$ gives $\hat{\lambda}(\psi)=1.10-(1.2/4)(4.20-\psi)$. The profiled log likelihood, apart from a constant, is

\ell_p(\psi) = -\frac{10}{2}\frac{(4.20-\psi)^2}{4}.

The unrestricted estimate of $\psi$ is 4.20. Hence $LR(\psi)=10(4.20-\psi)^2/4$. Using the 95% cutoff, the condition is $10(4.20-\psi)^2/4\leq3.841$, or $|4.20-\psi|\leq\sqrt{3.841(4/10)}\approx1.240$.

The interval is therefore approximately $[2.960,\,5.440]$ in the units of the first mean. At the lower endpoint, the nuisance estimate is $1.10-0.3(1.240)\approx0.728$; at the upper endpoint it is $1.10+0.3(1.240)\approx1.472$. The likelihood-ratio interval is symmetric here because this particular profile is quadratic, not because profile-likelihood intervals must be symmetric. With the known normal covariance and this regular model, the chi-square calibration is exact for this likelihood-ratio statistic.

Worked Example: Profiling for a Positive Transformed Target

Suppose the parameter of interest is the positive quantity $\rho=\exp(\psi)$, while $\lambda$ remains a nuisance mean. In a separate hypothetical sample of $n=25$ independent bivariate normal observations, assume the known covariance matrix has first diagonal entry 1, and the observed first sample mean is $\bar{x}_1=0.400$. The other mean is unrestricted. These assumptions give the same regular normal-mean likelihood structure as in the preceding example; the target is now a transformation of the first mean.

For a fixed $\rho>0$, the constraint is $\psi=\log(\rho)$. After profiling out $\lambda$, the likelihood-ratio statistic is

LR(\rho)=25\{\log(\rho)-0.400\}^2.

The 95% condition becomes $25\{\log(\rho)-0.400\}^2\leq3.841$, so $\log(\rho)$ lies between $0.400-\sqrt{3.841/25}$ and $0.400+\sqrt{3.841/25}$. Since $\sqrt{3.841/25}\approx0.392$, the interval on the log scale is approximately $[0.008,\,0.792]$. Exponentiating both endpoints gives the interval $[\exp(0.008),\exp(0.792)]\approx[1.008,\,2.208]$ for $\rho$.

The estimate is $\exp(0.400)\approx1.492$, and the interval is asymmetric around it on the original scale. Exponentiation preserves the ordering of endpoints because it is strictly increasing. This calculation also illustrates why the target’s scale matters: an interval symmetric in $\psi$ generally becomes asymmetric in $\exp(\psi)$.

Interpreting the Interval and Avoiding Common Errors

The endpoint equations compare each constrained maximum with the same unrestricted maximum. If an endpoint is a solution of $LR(\psi)=3.841$, then its profile log likelihood is lower than the unrestricted maximum by $3.841/2\approx1.921$. Values with a smaller likelihood-ratio statistic are retained; values with a larger one are excluded by this approximate test inversion.

A common mistake is to hold the nuisance parameter at its unrestricted estimate and evaluate the likelihood at each candidate target. That calculation compares the wrong constrained fit: the candidate value may require a different best-fitting nuisance value. The correction is to maximize over nuisance parameters anew for every candidate target, as in the Poisson example. Another mistake is to use the number of nuisance parameters as the chi-square degrees of freedom. For the regular test of one scalar restriction, the asymptotic cutoff uses one degree of freedom, regardless of how many nuisance parameters are estimated.

A profile interval is not automatically symmetric, and its center need not equal the target’s maximum-likelihood estimate. Do not replace its endpoints by an estimate plus or minus a standard error unless that separate approximation is specifically intended. Also, the chi-square result is not a universal finite-sample guarantee: non-identification, boundary values, or other failures of regularity can invalidate the usual calibration. The confidence interpretation is approximate under the stated conditions: across repeated samples, the procedure covers the true scalar target at a rate near $1-\alpha$.

Key takeaway: A scalar profile-likelihood confidence set contains the target values whose re-optimized likelihoods are within the one-degree-of-freedom chi-square cutoff of the unrestricted maximum. Re-optimizing nuisance parameters is essential, and the resulting endpoints can be asymmetric on the target’s scale.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In a model with a scalar target and four nuisance parameters, what degrees of freedom determine the usual asymptotic likelihood-ratio cutoff for testing a fixed target value? Explain.
Why must the nuisance parameter be re-estimated when constructing the Poisson rate-ratio profile?
For a 95% interval based on one scalar restriction, what likelihood-ratio statistic cutoff is used, and which candidate values are retained?
Why can a profile-likelihood interval be asymmetric even when it is centered, on another scale, at a maximum-likelihood estimate?
In the transformed-target example, why is it valid to exponentiate the two endpoints of the interval for $\psi$ to obtain an interval for $\rho=\exp(\psi)$?

Profile-Likelihood Confidence Intervals for Scalar Targets

What You'll Learn