Profile Score and Curvature

Why profile curvature differs from fixed-nuisance curvature

In a selection model, a selection parameter may be strongly entangled with parameters governing the outcome distribution. Profiling handles this by fixing the selection parameter in turn and re-optimizing the nuisance parameters at every fixed value. The resulting profile log likelihood describes how well each selection-parameter value can fit after the nuisance parameters have adapted. Its slope and curvature therefore need not match the corresponding derivatives with nuisance parameters held fixed.

This distinction matters because curvature measures local information. If nuisance parameters can compensate for a change in the selection parameter, the profile may be much flatter than the log likelihood looks when those nuisance parameters are artificially held still. The calculations below make that adjustment explicit.

Setup and regularity conditions

Let \(\theta\) be a scalar selection parameter and let \(\eta\) be a vector of nuisance parameters. Write the observed-data log likelihood as \(\ell(\theta,\eta)\). For each fixed \(\theta\), define \(\hat{\eta}(\theta)\) as a local maximizer over \(\eta\), and define the profile log likelihood by \(\ell_p(\theta)=\ell(\theta,\hat{\eta}(\theta))\). The hats here indicate optimization at the specified \(\theta\); they do not by themselves mean that \(\theta\) is at its joint maximum-likelihood estimate.

The formulas in this tutorial apply locally under the following conditions: \(\ell\) has continuous second derivatives near the point of interest; the nuisance optimum is an interior point; \(\ell_\eta(\theta,\hat{\eta}(\theta))=0\); and the nuisance Hessian \(\ell_{\eta\eta}\) there is nonsingular. For the nuisance point to be a strict local maximum, \(\ell_{\eta\eta}\) is negative definite. A unique smooth maximizing branch is also needed in the neighborhood considered. If there are constraints, a boundary optimum, or switching between distinct local maxima, the ordinary derivative formulas may not apply across that feature.

Subscripts denote partial derivatives: \(\ell_\theta\) is the derivative in \(\theta\) holding \(\eta\) fixed, \(\ell_{\eta\eta}\) is the nuisance Hessian, and \(\ell_{\theta\eta}\) is a row vector. All are evaluated at \((\theta,\hat{\eta}(\theta))\) when used in profile formulas.

Deriving the score and curvature

The profile log likelihood changes both because \(\theta\) changes and because the maximizing nuisance value moves. Apply the chain rule to its definition:

The nuisance score is zero at the interior optimum for each fixed \(\theta\). The second term therefore vanishes, giving the profile score:

This is the envelope property: for the first derivative, the change in the optimizing nuisance parameters contributes no additional term. It does not mean nuisance estimation is irrelevant to curvature.

To find how the nuisance optimum moves, differentiate its defining equation \(\ell_\eta(\theta,\hat{\eta}(\theta))=0\) with respect to \(\theta\). The chain rule gives

Now differentiate the profile score, including the movement of \(\hat{\eta}(\theta)\):

This is the profile curvature, also called the Schur-complement curvature. The formula is local: every derivative on its right-hand side is evaluated at the nuisance optimum for the current \(\theta\).

An equivalent expression uses observed information blocks. Define \(J=-\nabla^2\ell\), with blocks \(J_{\theta\theta}=-\ell_{\theta\theta}\), \(J_{\theta\eta}=-\ell_{\theta\eta}\), and \(J_{\eta\eta}=-\ell_{\eta\eta}\). Then

When \(J_{\eta\eta}\) is positive definite, the subtracted term is nonnegative because it is a quadratic form. Thus, under this condition, allowing the nuisance parameters to re-optimize cannot increase local information about \(\theta\) relative to holding them fixed. Strong cross-information between \(\theta\) and \(\eta\) can make the reduction substantial.

Several nuisance parameters

With a vector of nuisance parameters, the same result applies using matrices. The cross-information block describes how changes in \(\theta\) are linked to the nuisance parameters jointly; the matrix inverse accounts for their curvature and dependence on one another. The scalar subtraction in the first example becomes a quadratic form.

Diagnostics, alternatives, and limitations

A useful check is to compare analytic curvature with the curvature of a numerically computed profile. At nearby values \(\theta-h,\theta,\theta+h\), re-optimize \(\eta\) each time, then compare the analytic result with \[ \frac{\ell_p(\theta+h)-2\ell_p(\theta)+\ell_p(\theta-h)}{h^2}. \] Agreement should improve over a reasonable range of step sizes, subject to numerical error. If the result changes sharply with \(h\), or the fitted nuisance values jump between evaluations, the profile may be too noisy or the optimizer may have switched branches. Integration error in a selection-model likelihood can also contaminate derivatives, so a stable optimization alone is not sufficient evidence that the curvature is accurate.

A common mistake is to report \(J_{\theta\theta}\) as the information for \(\theta\) after nuisance parameters have been estimated. That is the curvature with \(\eta\) held fixed, not the profile curvature. The correction is the Schur-complement term \(J_{\theta\eta}J_{\eta\eta}^{-1}J_{\eta\theta}\). Another mistake is to assume that the profile is always less curved. That conclusion requires a positive definite nuisance information block; if it is singular, nearly singular, or not positive definite, the inverse-based calculation can fail or be unstable.

The main alternative is joint curvature of the full log likelihood, which retains the entire parameter block rather than reducing it to a curve in \(\theta\). That is useful for understanding parameter dependence, but it does not replace profiling when the question concerns how the best attainable fit changes as \(\theta\) varies. Holding nuisance parameters fixed is simpler but answers a different question. Profile curvature is also local: it does not describe distant branches, boundary behavior, or the global shape of the likelihood. In particular, a small profile information value signals weak local curvature, not by itself a complete diagnosis of why the fit is unstable.

At a joint interior maximum, the profile score is zero because the partial score \(\ell_\theta\) is zero there. Away from that maximum, the profile score still equals the partial score evaluated at the re-optimized nuisance values. This distinction is useful when checking a calculation: the nuisance score should be zero along the profile, but the selection-parameter score need not be zero except at a stationary point of the profile.