Why Profile the Likelihood?
A model may have many parameters, while the scientific question concerns only one or a few. The remaining parameters are nuisance parameters: they matter for describing the model, but they are not the target of inference. Profile likelihood retains their effect by optimizing them for each possible value of the target, producing a likelihood on the target parameter alone.
This reduction is useful when a full multivariate approximation obscures the uncertainty in the quantity of interest. The shape of the profile near its maximum gives a local measure of uncertainty; comparing its values across the parameter space gives likelihood-based confidence sets. Profiling is optimization, not integration and not conditioning.
Definition and Basic Setup
Write the full parameter as \(\theta=(\psi,\lambda)\), where \(\psi\in\mathbb{R}^q\) is the parameter of interest and \(\lambda\in\mathbb{R}^r\) is a nuisance parameter. Let \(\ell(\psi,\lambda)\) be the log-likelihood, and let \(\Lambda(\psi)\) be the allowable nuisance-parameter values when \(\psi\) is fixed. The parameter space may impose constraints, so \(\Lambda(\psi)\) need not be the same for every \(\psi\).
If the full maximum-likelihood estimator is \((\hat{\psi},\hat{\lambda})\), and the optimization is well behaved, then \(\hat{\psi}\) maximizes \(\ell_p\). The profile is generally not itself the likelihood of a new sampling model for \(\psi\); it is a reduced function derived from the original likelihood.
Worked Example: Profiling the Mean with Unknown Variance
Suppose \(X_1,\ldots,X_n\) are independent \(N(\mu,\sigma^2)\) observations, with \(\mu\) the target and \(\sigma^2>0\) nuisance. For observed data, write \(\bar{x}\) for the sample mean and \(S=\sum_{i=1}^n(x_i-\bar{x})^2\). The log-likelihood, omitting a constant independent of the parameters, is \(\ell(\mu,\sigma^2)=-\frac n2\log(\sigma^2)-\frac{S+n(\bar{x}-\mu)^2}{2\sigma^2}\).
For fixed \(\mu\), set \(A(\mu)=S+n(\bar{x}-\mu)^2\). Differentiating with respect to \(\sigma^2\) gives \(-n/(2\sigma^2)+A(\mu)/(2(\sigma^2)^2)\), which vanishes at \(\hat{\sigma}^2_\mu=A(\mu)/n\). Substitution yields, up to a constant, \(\ell_p(\mu)=-\frac n2\log\{[S+n(\bar{x}-\mu)^2]/n\}\).
Take \(n=5\), \(\bar{x}=10\), and \(S=20\). The profile is maximized at \(\hat{\mu}=10\), with \(\hat{\sigma}^2_{\hat{\mu}}=20/5=4\). Relative to its maximum, \(2\{\ell_p(10)-\ell_p(\mu)\}=5\log\{1+5(\mu-10)^2/20\}\).
Using the usual 95% one-parameter likelihood-ratio cutoff \(3.841\), the endpoints satisfy \(5\log\{1+(\mu-10)^2/4\}=3.841\). Thus \((\mu-10)^2=4\{\exp(3.841/5)-1\}\), giving \(|\mu-10|\approx2.15\). The approximate likelihood interval is \([7.85,12.15]\). Its cutoff is asymptotic; it is not an exact finite-sample coverage claim.
Profile Curvature and the Schur Complement
Assume that the full maximum is an interior point and that \(\ell\) is twice continuously differentiable near it. Suppose also that for target values near \(\hat{\psi}\), the nuisance maximizer \(\hat{\lambda}_{\psi}\) is unique, interior, and differentiable, with the nuisance Hessian nonsingular. Define the observed information at the full maximum by \(H=-\nabla^2\ell(\hat{\psi},\hat{\lambda})\), and partition it into blocks:
Here \(H_{\lambda\lambda}\) must be invertible. For a strict local maximum it is positive definite. The profile curvature accounts for the fact that the best-fitting nuisance parameter changes as \(\psi\) changes.
Proof. At an interior nuisance maximum, \(\ell_\lambda(\psi,\hat{\lambda}_{\psi})=0\). Differentiate this equation with respect to \(\psi\). Using the matrix convention that derivatives are arranged as Jacobian blocks gives \(\ell_{\lambda\psi}+\ell_{\lambda\lambda}\,d\hat{\lambda}_{\psi}/d\psi=0\). Since \(\ell_{\lambda\lambda}=-H_{\lambda\lambda}\) and \(\ell_{\lambda\psi}=-H_{\lambda\psi}\), it follows that \(d\hat{\lambda}_{\psi}/d\psi=-H_{\lambda\lambda}^{-1}H_{\lambda\psi}\) at the maximum.
By the chain rule, the profile score is \(\ell_{p,\psi}=\ell_\psi+\ell_\lambda\,d\hat{\lambda}_{\psi}/d\psi=\ell_\psi\), because \(\ell_\lambda=0\) along the nuisance maximum. Differentiating the profile score once more and evaluating at the full maximum gives \(\nabla^2\ell_p=\ell_{\psi\psi}+\ell_{\psi\lambda}\,d\hat{\lambda}_{\psi}/d\psi\). Substitute \(\ell_{\psi\psi}=-H_{\psi\psi}\), \(\ell_{\psi\lambda}=-H_{\psi\lambda}\), and the derivative just obtained. Then \(\nabla^2\ell_p=-H_{\psi\psi}+H_{\psi\lambda}H_{\lambda\lambda}^{-1}H_{\lambda\psi}\), which proves the stated formula. \(\square\)
This identity also explains why the target’s uncertainty cannot generally be read from \(H_{\psi\psi}\) alone. Allowing the nuisance parameter to adjust reduces the effective curvature relative to holding it fixed. In a regular model, the block-inverse identity gives \((H^{-1})_{\psi\psi}=H_p^{-1}\). Thus the target block of the full covariance approximation agrees with the inverse profile curvature.
Worked Example: Correlated Target and Nuisance Estimates
Consider a likelihood that is exactly quadratic near its maximum: \(\ell(\psi,\lambda)=C-\frac12\{4(\psi-1)^2+4(\psi-1)(\lambda+1)+9(\lambda+1)^2\}\). Its information matrix is \(\begin{pmatrix}4&2\\2&9\end{pmatrix}\), which is positive definite since its leading diagonal entry is positive and its determinant is \(36-4=32>0\). The full maximum is \((1,-1)\).
For fixed \(\psi\), differentiating with respect to \(\lambda\) gives \(-2(\psi-1)-9(\lambda+1)=0\), so \(\hat{\lambda}_\psi=-1-\frac29(\psi-1)\). Substituting this into the quadratic, or using the Schur complement, gives \(H_p=4-\frac{2\cdot2}{9}=\frac{32}{9}\).
Therefore the profile is \(\ell_p(\psi)=C-\frac12\frac{32}{9}(\psi-1)^2\), and the curvature-based standard error for \(\hat{\psi}\) is \(1/\sqrt{32/9}=3/\sqrt{32}\approx0.530\). Using \(1.96\) standard errors gives the approximate interval \(1\pm1.96(0.530)\), or approximately \([-0.04,2.04]\). The nuisance adjustment changes the curvature from \(4\) to \(32/9\); ignoring it would understate the standard error.
Local Quadratic Approximation and Inference
For asymptotic inference, suppose observations are independent and identically distributed under a correctly specified, regular model. Assume the true parameter is interior, the relevant information is nonsingular, the maximum-likelihood estimator is consistent and asymptotically normal, and the required derivatives and likelihood expansions hold locally. Let \(J\) denote the limiting information per observation at the true parameter, partitioned into the same target and nuisance blocks. The profile’s effective information per observation is \(J_{\mathrm{eff}}=J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi}\).
The local quadratic approximation follows by expanding the full log-likelihood around its maximum. If \(h=\psi-\hat{\psi}\), and \(k=\lambda-\hat{\lambda}\), its leading change is \(-\frac n2(h^\mathsf{T}J_{\psi\psi}h+2h^\mathsf{T}J_{\psi\lambda}k+k^\mathsf{T}J_{\lambda\lambda}k)\). For fixed \(h\), this quadratic is maximized over \(k\) at \(k=-J_{\lambda\lambda}^{-1}J_{\lambda\psi}h\): differentiating the expression in parentheses with respect to \(k\) gives \(2J_{\lambda\psi}h+2J_{\lambda\lambda}k=0\). Substituting this value yields the profile loss \(-\frac n2 h^\mathsf{T}J_{\mathrm{eff}}h\). The Taylor remainder is \(o_p(1)\) for local displacements \(h,k=O_p(n^{-1/2})\) under the stated regularity conditions. Hence, locally, \(\ell_p(\hat{\psi}+h)-\ell_p(\hat{\psi})=-\frac n2h^\mathsf{T}J_{\mathrm{eff}}h+o_p(1)\).
In practice, the observed profile curvature \(-\nabla^2\ell_p(\hat{\psi})\) gives the curvature-based covariance estimate for \(\hat{\psi}\); for a scalar target, the standard error is the reciprocal square root of that curvature. For a \(q\)-dimensional target, the corresponding covariance estimate is the inverse curvature matrix. The likelihood-based confidence set uses a likelihood-ratio cutoff: \(2\{\ell_p(\hat{\psi})-\ell_p(\psi)\}\leq\chi^2_{q,1-\alpha}\), where \(\chi^2_{q,1-\alpha}\) is the \(1-\alpha\) quantile of a chi-square distribution with \(q\) degrees of freedom. Under the regularity conditions of Wilks’ theorem, this set has asymptotic coverage \(1-\alpha\). Its shape need not be symmetric, even though its local approximation is quadratic.
Worked Example: Profiling a Log Rate Ratio
Let \(X_1\sim\operatorname{Poisson}(\lambda_1)\) and \(X_2\sim\operatorname{Poisson}(\lambda_2)\) independently, with observed counts \(x_1=8\) and \(x_2=2\). The target is the log rate ratio \(\psi=\log(\lambda_1/\lambda_2)\). Parameterize the nuisance level by \(\eta\), so that \(\lambda_1=e^{\eta+\psi/2}\) and \(\lambda_2=e^{\eta-\psi/2}\). Up to constants, the log-likelihood is \(\ell(\psi,\eta)=10\eta+3\psi-2e^\eta\cosh(\psi/2)\).
For fixed \(\psi\), the \(\eta\)-score is \(10-2e^\eta\cosh(\psi/2)\). Setting it to zero gives \(e^{\hat{\eta}_\psi}=5/\cosh(\psi/2)\). Substitution gives, up to a constant independent of \(\psi\), \(\ell_p(\psi)=3\psi-10\log\cosh(\psi/2)\).
The profile score is \(3-5\tanh(\psi/2)\). Its zero satisfies \(\tanh(\hat{\psi}/2)=3/5\), so \(\hat{\psi}=2\operatorname{atanh}(3/5)=\log 4\), matching the observed log rate ratio. The second derivative is \(-\frac52\operatorname{sech}^2(\psi/2)\). At \(\hat{\psi}=\log4\), \(\tanh(\hat{\psi}/2)=3/5\), so \(\operatorname{sech}^2(\hat{\psi}/2)=1-9/25=16/25\). The profile information is therefore \(\frac52(16/25)=1.6\), and the curvature-based standard error is \(1/\sqrt{1.6}\approx0.791\).
A 95% likelihood interval is obtained by solving \(2\{\ell_p(\log4)-\ell_p(\psi)\}\leq3.841\). This retains the exact profile shape in the interval calculation; the curvature-based interval \(\log4\pm1.96(0.791)\) is instead a local symmetric approximation.
Common Pitfall: Fixing or Integrating the Nuisance Parameter
A common mistake is to calculate uncertainty for \(\psi\) from \(H_{\psi\psi}\) as if \(\lambda\) were known. That uses the curvature with the nuisance parameter held fixed. In the quadratic example, it would give standard error \(1/\sqrt4=0.5\), instead of the profile-based value \(1/\sqrt{32/9}\approx0.530\). Profiling allows the nuisance parameter to respond to changes in \(\psi\), so the correct curvature includes the Schur-complement adjustment.
Another mistake is to treat profiling as marginalization. Maximizing over \(\lambda\) selects its best-fitting value at each \(\psi\); marginalization instead integrates over nuisance values with respect to a specified measure or prior. These procedures answer different questions and need not give the same function of \(\psi\). Profile likelihood is most directly useful when the likelihood is regular and the nuisance maximizer is well behaved. Boundaries, nonunique maxima, or singular information can invalidate the smooth curvature approximation and require separate analysis.
Check Your Understanding
Use the definitions and calculations above to answer the following.
- In one sentence, distinguish profiling over a nuisance parameter from integrating over it.
- Why is the profile curvature generally not just \(H_{\psi\psi}\)?
- For a scalar target, how is a curvature-based standard error obtained from the profile log-likelihood?
- What degrees of freedom appear in the asymptotic likelihood-ratio cutoff for a \(q\)-dimensional target?
- In the Poisson example, what equation determines the nuisance estimate for a fixed value of \(\psi\)?