Profile Likelihood for Generalized Model Parameters

Why Profile a Likelihood?

A generalized model may contain many regression coefficients, while the scientific question concerns just one of them or a particular contrast among them. Wald inference summarizes uncertainty through an estimate and its standard error. Profile likelihood instead asks how well each proposed value of the target parameter can fit the data after the other coefficients have been chosen as favorably as possible. This gives a likelihood curve on the parameter of interest's own scale and can reveal asymmetry that a symmetric Wald interval does not show.

The construction is a constrained optimization problem. Fix the parameter value being assessed, maximize the likelihood over all remaining coefficients, and compare that constrained maximum with the unrestricted maximum. The nuisance coefficients are not set to zero or held at their unrestricted estimates: they are re-estimated for every fixed target value.

Definition and Construction

Definition: Let $\theta=(\psi,\lambda)$ parameterize a model, where $\psi$ is a scalar parameter of interest and $\lambda$ is a vector of nuisance parameters. For observed data $y$, the profile log-likelihood for $\psi$ is $$ \ell_p(\psi)=\sup_{\lambda\in\Lambda(\psi)}\ell(\psi,\lambda), $$ where $\Lambda(\psi)$ is the set of nuisance-parameter values allowed when $\psi$ is fixed. The profile likelihood is $L_p(\psi)=\exp\{\ell_p(\psi)\}$. If the unrestricted maximum exists, write $\hat\ell=\sup_{\psi,\lambda}\ell(\psi,\lambda)$; the profile likelihood-ratio statistic at $\psi_0$ is $$ W(\psi_0)=2\{\hat\ell-\ell_p(\psi_0)\}. $$

For generalized linear models with coefficient vector $\beta$, the parameter of interest may be a coefficient or a linear contrast $c^T\beta$, where $c$ is fixed and nonzero. In the contrast case, choose an invertible linear reparameterization whose first coordinate is $\psi=c^T\beta$; the other coordinates form $\lambda$. Maximizing in the new nuisance coordinates is exactly maximizing over the original coefficient vectors that satisfy $c^T\beta=\psi$.

The likelihood-ratio curve is $W(\psi)=2\{\hat\ell-\ell_p(\psi)\}$. It is nonnegative because the unrestricted parameter space includes every constrained submodel. A conventional large-sample $100(1-\alpha)\%$ confidence set consists of parameter values whose statistic does not exceed the $1-\alpha$ quantile of a chi-square distribution with one degree of freedom:

\mathcal C_{1-\alpha} = \left\{\psi: W(\psi)\leq \chi^2_{1,1-\alpha}\right\}.

This calibration is asymptotic, not generally an exact finite-sample guarantee. It requires a regular interior parameter point, an identifiable model, sufficient smoothness, nonsingular information, and a well-behaved maximum-likelihood estimate. A boundary value, a nonexistent or infinite estimate, or a nuisance maximum that fails to exist can invalidate the usual chi-square approximation.

What the Profile Curve Measures

Theorem: Suppose the unrestricted likelihood maximum exists and, for each considered $\psi$, the constrained supremum over $\lambda$ exists. Then $W(\psi)$ is exactly the likelihood-ratio statistic comparing the unrestricted model with the model restricted by $\psi$'s fixed value.

To verify this, let $\ell_{\mathrm{full}}=\sup_{\psi,\lambda}\ell(\psi,\lambda)$ and $\ell_{\mathrm{con}}(\psi)=\sup_{\lambda}\ell(\psi,\lambda)$. The likelihood-ratio statistic for these two models is, by definition, twice the difference of their maximized log-likelihoods:

2\{\ell_{\mathrm{full}}-\ell_{\mathrm{con}}(\psi)\} = 2\{\hat\ell-\ell_p(\psi)\} = W(\psi).

Thus profiling does not introduce a different test statistic. It computes the likelihood-ratio comparison separately at each possible value of the scalar parameter. If the restricted model is nested in a regular unrestricted model and $\psi$ is scalar, Wilks' theorem gives the chi-square-one reference distribution under the null, asymptotically.

A related fact helps interpret the curve locally. Suppose the constrained maximizer $\hat\lambda(\psi)$ is an interior point, is unique in a neighborhood, and is differentiable in $\psi$. Then the profile score is

\frac{d}{d\psi}\ell_p(\psi) = \frac{\partial}{\partial\psi}\ell\{\psi,\hat\lambda(\psi)\}.

Indeed, applying the chain rule gives the partial derivative with respect to $\psi$ plus the nuisance score multiplied by $d\hat\lambda/d\psi$. At the constrained maximum the nuisance score is zero, so the second term vanishes. This is the envelope principle. It does not say that the nuisance estimates stay fixed: they can change with $\psi$; their effect disappears from the first derivative because each is re-optimized.

Worked Examples

Worked Example: A Poisson Regression Coefficient

Suppose $Y_0$ and $Y_1$ are independent Poisson counts with means $\mu_0=\exp(\alpha)$ and $\mu_1=\exp(\alpha+\beta)$, and the observed counts are $y_0=4$ and $y_1=9$. Find the profile log-likelihood for the slope $\beta$, and assess whether $\beta=0$ lies in its asymptotic 95% likelihood-ratio confidence set.

Ignoring terms constant in the parameters, the log-likelihood is

\ell(\alpha,\beta) = 4\alpha-\exp(\alpha) + 9(\alpha+\beta)-\exp(\alpha+\beta).

For fixed $\beta$, the score in $\alpha$ is $13-\exp(\alpha)(1+\exp(\beta))$. Setting it to zero gives

\hat\alpha(\beta) = \log\left\{\frac{13}{1+\exp(\beta)}\right\}.

The fitted means under this constraint are $13/(1+\exp(\beta))$ and $13\exp(\beta)/(1+\exp(\beta))$. The profile score, either by substitution or the envelope result, is $9-13\exp(\beta)/(1+\exp(\beta))$. It vanishes at $\exp(\hat\beta)=9/4$, so $\hat\beta=\log(9/4)$; the corresponding means are 4 and 9.

At $\beta=0$, the constrained fit has a common mean $13/2=6.5$. The likelihood-ratio statistic is therefore

W(0) = 2\left\{ 4\log\left(\frac{4}{6.5}\right) + 9\log\left(\frac{9}{6.5}\right) \right\} \approx 1.974.

Since $1.974<\chi^2_{1,0.95}\approx3.841$, the value zero is in the asymptotic 95% likelihood-ratio confidence set. The conclusion comes from the profile comparison, not from holding the intercept at its unrestricted estimate when testing the slope.

Worked Example: Profiling a Slope in a Gaussian Generalized Model

Let $Y_i$ be independent $N(\alpha+\beta x_i,1)$ observations at $x=(-1,0,1)$, with observed values $y=(1,2,4)$. Treat the variance as known. Construct the profile likelihood-ratio confidence set for the slope $\beta$.

For any fixed $\beta$, the maximizing intercept is the average of $y_i-\beta x_i$. Because $\sum_i x_i=0$, this is $\hat\alpha(\beta)=7/3$. The unrestricted slope estimate is $\hat\beta=\sum_i x_i y_i/\sum_i x_i^2=3/2$. The residual sum of squares at the unrestricted fit is

(1-5/6)^2+(2-7/3)^2+(4-23/6)^2 = \frac{1}{36}+\frac{4}{36}+\frac{1}{36} = \frac{1}{6}.

At any fixed $\beta$, the residual sum of squares is $1/6+2(\beta-3/2)^2$: the change in slope contributes $(\beta-3/2)^2\sum_i x_i^2=2(\beta-3/2)^2$, and the intercept remains the least-squares choice. With known unit variance, twice the log-likelihood drop equals the increase in residual sum of squares. Consequently,

W(\beta)=2(\beta-3/2)^2.

The asymptotic 95% set is $2(\beta-3/2)^2\leq3.841$, or $\beta\in[3/2-\sqrt{3.841/2},\,3/2+\sqrt{3.841/2}]\approx[0.114,2.886]$. Here the profile curve is exactly quadratic. In many non-Gaussian models it is not, and its confidence set need not be symmetric around the maximum-likelihood estimate.

Worked Example: A Logistic Treatment Effect

In a two-group binomial logistic model, let $\alpha$ be the control log-odds and $\beta$ the treatment log-odds contrast. There are 10 observations per group, with 1 success in the control group and 5 in the treatment group. Test $\beta=0$ by profiling over $\alpha$.

Writing $p_0=\operatorname{expit}(\alpha)$ and $p_1=\operatorname{expit}(\alpha+\beta)$, the parameter-dependent log-likelihood is

\ell(\alpha,\beta) = \log(p_0)+9\log(1-p_0) + 5\log(p_1)+5\log(1-p_1).

At fixed $\beta$, the intercept score equation is $1+5-10p_0-10p_1=0$, so the nuisance fit must satisfy $p_0+p_1=0.6$. This equation, together with the logistic relation between $p_0$ and $p_1$, determines the constrained fit for every proposed $\beta$. At the unrestricted maximum, the fitted proportions are $p_0=0.1$ and $p_1=0.5$, giving $\hat\alpha=\log(1/9)$ and $\hat\beta=\log(9)$.

Under the null constraint $\beta=0$, both groups have a common fitted proportion $p=6/20=0.3$. The likelihood-ratio statistic is

\begin{aligned} W(0) &= 2\left\{ 1\log\left(\frac{0.1}{0.3}\right) +9\log\left(\frac{0.9}{0.7}\right) +5\log\left(\frac{0.5}{0.3}\right) +5\log\left(\frac{0.5}{0.7}\right) \right\}\\ &\approx 4.070. \end{aligned}

Because $4.070>3.841$, zero is excluded from the asymptotic 95% likelihood-ratio confidence set. The profile curve for other values of $\beta$ is obtained by solving the intercept score equation at each value and comparing that fit with the unrestricted maximum.

Why the Chi-Square Calibration Works

The one-degree-of-freedom calibration follows from the local quadratic behavior of a regular likelihood. The following derivation states the conditions that matter. Suppose the true parameter is an interior point $\theta_0=(\psi_0,\lambda_0)$, the model is identifiable and differentiable, the Fisher information per observation $I$ is positive definite, and the likelihood has a uniform local quadratic expansion. Assume also that the maximum-likelihood estimates are consistent and asymptotically normal. These are standard regularity conditions for likelihood inference; they exclude boundary points and singular information.

Theorem: Under these regularity conditions, for testing the scalar restriction $\psi=\psi_0$, the profile likelihood-ratio statistic satisfies $W(\psi_0)\xrightarrow{d}\chi^2_1$.

For a sample of size $n$, write $h=\sqrt n(\theta-\theta_0)$, and partition $h=(h_\psi,h_\lambda)$, the score limit $\Delta_n=(\Delta_{\psi,n},\Delta_{\lambda,n})$, and the information matrix $I$ in the same way. The local quadratic expansion is

\ell(\theta_0+h/\sqrt n)-\ell(\theta_0) = h^T\Delta_n-\frac12h^TIh+o_p(1), \qquad \Delta_n\xrightarrow{d}N(0,I).

The unrestricted maximum of this quadratic occurs at $h=I^{-1}\Delta_n$. Under the restriction $\psi=\psi_0$, the local coordinate $h_\psi$ is zero. Maximizing over $h_\lambda$ gives $h_\lambda=I_{\lambda\lambda}^{-1}\Delta_{\lambda,n}$. Subtracting the constrained quadratic maximum from the unrestricted one, or completing the square in the $\psi$ coordinate, yields

W(\psi_0) = \frac{ \left( \Delta_{\psi,n} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}\Delta_{\lambda,n} \right)^2 }{ I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} } +o_p(1).

The denominator is the positive Schur complement of $I_{\lambda\lambda}$ in $I$. The numerator's unsquared term converges to a mean-zero normal variable with variance equal to that same Schur complement: this follows by applying the covariance $I$ to the displayed linear combination of the score coordinates. Dividing by the square root of that variance gives a standard normal limit. Its square has a $\chi^2_1$ distribution, proving the result. The error is negligible by the assumed uniform local expansion and consistency of the unrestricted and restricted maximizers. $\square$

This argument also explains why nuisance parameters do not add degrees of freedom to the reference distribution: they are freely optimized under both models. The restriction removes one local parameter dimension, so the likelihood-ratio statistic has one degree of freedom. For $r$ independent regular scalar restrictions, the corresponding dimension difference is $r$, but the present confidence-set construction concerns one scalar target.

Practical Computation and Common Pitfalls

In a numerical generalized-model analysis, compute the unrestricted maximum once, then repeat the constrained nuisance optimization over a grid of $\psi$ values. For each value, store the maximized log-likelihood and calculate $W(\psi)$. The confidence set is the part of the profile curve below the chosen chi-square cutoff. A root-finding method can locate its endpoints when the curve crosses that cutoff. Check optimizer convergence at each grid point, use multiple starting values if the likelihood may have multiple local maxima, and verify that the reported nuisance solution satisfies the constraint.

Choose the target.
Specify a scalar coefficient or contrast $\psi=c^T\beta$, then express the remaining free coordinates as nuisance parameters.

Fit the unrestricted model.
Find $\hat\ell$, the maximized log-likelihood over all coefficients.

Profile each proposed value.
Fix $\psi$, maximize over the nuisance coefficients, and record $\ell_p(\psi)$.

Form and interpret the curve.
Calculate $W(\psi)=2\{\hat\ell-\ell_p(\psi)\}$; retain values below the appropriate chi-square cutoff when regular asymptotic inference is justified.

Common mistake: A researcher fixes the target coefficient at $\psi_0$, leaves the nuisance coefficients at their unrestricted estimates, and calls the resulting likelihood difference the profile likelihood-ratio statistic. That calculation does not compare the unrestricted model with the best-fitting model subject to $\psi=\psi_0$. The correction is to re-maximize over all nuisance coefficients for every fixed $\psi$. Another error is to treat the likelihood-ratio confidence set as an exact finite-sample interval; its usual chi-square cutoff is justified by regular large-sample theory, not by the definition of profiling itself.

Profiling is valuable when the likelihood is noticeably asymmetric, when the parameter is a contrast whose uncertainty is not well summarized by a symmetric interval, or when a direct likelihood comparison is scientifically natural. It does not automatically cure poor model behavior: sparse data, separation, boundary estimates, and weak identification can make the curve irregular or prevent a finite maximum. In such cases, the computation and the calibration both require additional care.

Check Your Understanding

Answer these questions by referring to the definition and derivation.

For a scalar coefficient $\psi$ and nuisance vector $\lambda$, what is held fixed and what is maximized to obtain $\ell_p(\psi)$?
Explain why the profile likelihood-ratio statistic equals the likelihood-ratio statistic comparing a restricted and an unrestricted model.
How can a linear contrast $c^T\beta$ be used as the first coordinate of a reparameterized model?
What conditions are needed for the usual chi-square-one calibration, and name one situation that can violate them.
Why is it incorrect to keep nuisance coefficients fixed at their unrestricted estimates while profiling?