Curvature, Flat Profiles, and Weakly Determined Parameters

Why Curvature Matters

A likelihood maximizer can be numerically well defined while some combinations of its parameters are only weakly determined by the observed data. In that situation, the likelihood falls sharply in some directions but changes very little in others. A report of the maximizing parameter vector alone can hide this feature: substantially different vectors may have nearly indistinguishable likelihood values.

Local curvature, profile-likelihood shape, and sensitivity to optimization choices provide complementary diagnostics. They describe the shape of the likelihood near a fitted value and help distinguish a genuinely flat direction from problems caused by parameter units, numerical calculations, or an inadequate search. They do not, by themselves, settle whether a model is identified; that is a separate question.

Definition: Let \(\ell(\theta)\) be a log likelihood for a parameter vector \(\theta\), and let \(\hat{\theta}\) be an interior local maximizer. The observed information is \(J(\hat{\theta})=-\nabla^2\ell(\hat{\theta})\), the negative Hessian of the log likelihood at the fitted value. If \(J\) is positive definite, its eigenvectors give local parameter directions and its positive eigenvalues measure curvature in those directions. A small eigenvalue relative to the others indicates a locally weakly curved direction.

The eigenvalue interpretation follows from a local quadratic approximation. For a small displacement \(h\) from an interior stationary point,

\ell(\hat{\theta}+h)\approx \ell(\hat{\theta})-\frac{1}{2}h'J(\hat{\theta})h.

If \(v\) is a unit eigenvector of \(J\) with eigenvalue \(\lambda\), then a displacement \(h=t v\) gives an approximate log-likelihood loss of \(\frac{1}{2}\lambda t^2\). A small \(\lambda\) means that a larger displacement along \(v\) is needed to produce the same local loss. The eigenvector matters as much as the eigenvalue: a weak direction may involve a combination of several parameters rather than one parameter alone.

This interpretation requires care. Eigenvalues depend on how parameters are scaled: expressing a parameter in dollars rather than thousands of dollars changes the numerical curvature in its coordinate. A small eigenvalue is most informative after considering parameter units, a meaningful scaling, and the direction represented by its eigenvector. At a constrained boundary, a standard unconstrained local quadratic interpretation may not apply; the relevant parameter space and boundary behavior must be respected.

Worked Curvature Calculation

Worked Example: Finding a Nearly Flat Parameter Combination

Consider a hypothetical two-parameter log likelihood, defined up to an additive constant, by \(\ell(a,b)=-Q(a,b)\), where

Q(a,b)=\frac{1}{2}\left\{(a+b-3)^2+0.0002(a-b)^2\right\}.

Both squared terms are nonnegative and equal zero at \(a=b=1.5\), so the maximum occurs at \((\hat a,\hat b)=(1.5,1.5)\). The observed information is the Hessian of \(Q\). Differentiating the two terms gives

J= \begin{pmatrix} 1.0002&0.9998\\ 0.9998&1.0002 \end{pmatrix}.

For a symmetric matrix of this form, the vector \((1,1)'\) has eigenvalue \(1.0002+0.9998=2\), and \((1,-1)'\) has eigenvalue \(1.0002-0.9998=0.0004\). The corresponding unit eigenvectors are \(v_+=(1,1)'/\sqrt{2}\) and \(v_-=(1,-1)'/\sqrt{2}\). Thus the common movement of \(a\) and \(b\) is much more sharply curved than their contrast.

The factor of two in the small eigenvalue can also be checked directly. The difference term contributes \(0.0002\begin{pmatrix}1&-1\\-1&1\end{pmatrix}\) to the Hessian. This matrix has eigenvalue \(0.0004\) in the \((1,-1)'\) direction, not \(0.0002\). A unit displacement \(h=t v_-\) therefore produces the approximate log-likelihood loss \(\frac{1}{2}(0.0004)t^2=0.0002t^2\). A displacement of \(t=10\) in this direction loses only \(0.02\) log-likelihood units under this quadratic likelihood.

The example has positive curvature in both directions, so its likelihood is not exactly flat in the contrast direction. It is, however, very weakly curved there relative to the common direction. The parameters are strongly constrained as a sum, but a substantially larger range of contrasts gives similar fit.

Profile Likelihood Reveals Curvature for a Target

A Hessian eigenvector describes a direction in the full parameter space. When the question concerns a particular parameter, a profile likelihood gives a more targeted diagnostic. Partition the parameter vector into a scalar target \(\psi\) and nuisance parameters \(\lambda\). The profile log likelihood is the log likelihood maximized over the nuisance parameters at each fixed target value:

\ell_p(\psi)=\max_{\lambda}\ell(\psi,\lambda).

A broad, slowly declining profile indicates that a range of values for \(\psi\) can be accompanied by suitable nuisance-parameter values with little loss in fit. The full Hessian and a target's profile curvature need not tell the same simple story: profiling allows nuisance parameters to adjust, which can make the target less sharply determined than a calculation holding those parameters fixed would suggest.

Local profile-curvature result: At a smooth interior maximum, partition the observed information as \(J=\begin{pmatrix}J_{\psi\psi}&J_{\psi\lambda}\\J_{\lambda\psi}&J_{\lambda\lambda}\end{pmatrix}\). If \(J_{\lambda\lambda}\) is positive definite, the second derivative of the profiled negative log likelihood at the maximum is \(J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi}\). This is the Schur complement of the nuisance block.

To see the result, write \(Q=-\ell\) and use a local quadratic approximation around the maximum. Let \(u\) be the displacement in \(\psi\) and \(w\) the displacement in \(\lambda\). The quadratic increase in \(Q\) is

\frac{1}{2}\left(J_{\psi\psi}u^2+2uJ_{\psi\lambda}w+w'J_{\lambda\lambda}w\right).

For a fixed \(u\), profiling minimizes this expression over \(w\). Differentiating with respect to \(w\) and setting the derivative to zero gives \(J_{\lambda\lambda}w+J_{\lambda\psi}u=0\), so \(w=-J_{\lambda\lambda}^{-1}J_{\lambda\psi}u\). Substitution yields the quadratic increase \(\frac{1}{2}u^2(J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi})\), establishing the stated local curvature. The result is local and assumes a smooth interior nuisance optimum; it is not a general claim about profiles at boundaries or far from the maximum.

Worked Example: Computing Profile Curvature

Use the quadratic likelihood from the previous example and profile \(a\), optimizing over \(b\). Here \(a\) is the target and \(b\) is nuisance, so the local profile curvature is

J_{aa}-J_{ab}J_{bb}^{-1}J_{ba} =1.0002-\frac{(0.9998)^2}{1.0002} =\frac{(1.0002)^2-(0.9998)^2}{1.0002} =\frac{0.0008}{1.0002} \approx 0.0007998.

Thus the profiled negative log likelihood near \(a=1.5\) is approximately \(\frac{1}{2}(0.0007998)(a-1.5)^2\). For example, a displacement of \(a\) by 5 units gives an approximate increase of \(0.5(0.0007998)(25)\approx0.0100\) in \(Q\), or a log-likelihood decrease of about \(0.0100\). The nuisance estimate is allowed to adjust as \(a\) changes, which is why the profile curvature is smaller than the fixed-\(b\) curvature \(J_{aa}=1.0002\).

The small profile curvature signals that \(a\) is weakly determined when \(b\) is allowed to compensate. This calculation is a local diagnostic; the full profile should also be inspected to check whether the quadratic approximation remains informative over the range of interest.

Starting Values and Numerical Flatness

Optimization behavior can expose a flat likelihood, but it can also be misleading. If different starting values lead to noticeably different parameter estimates while the maximized log likelihoods are nearly identical, the data and likelihood may provide little local distinction among those parameter combinations. However, different solutions can also indicate a poor optimizer, a saddle point, a boundary solution, or numerical error. Compare gradients, likelihood values, constraints, and profile evaluations before drawing a conclusion.

Worked Example: Checking Sensitivity to Starting Values

For the hypothetical quadratic likelihood above, write \(d=a-b\) and restrict attention to the ridge \(a+b=3\). Then \(a=(3+d)/2\), \(b=(3-d)/2\), and

Q=\frac{1}{2}(0.0002d^2)=0.0001d^2.

The exact maximum on this ridge is at \(d=0\), but the likelihood falls slowly as \(d\) moves away from zero. Suppose an optimizer with a relatively loose stopping tolerance is run from several starting values and reports candidate points with \(d=1\) and \(d=5\), both on the ridge. At \(d=1\), \((a,b)=(2,1)\) and \(Q=0.0001\). At \(d=5\), \((a,b)=(4,-1)\) and \(Q=0.0025\). Since \(\ell=-Q\), the log-likelihood difference between these candidates is only \(0.0025-0.0001=0.0024\), despite the different parameter values.

These are not equal-likelihood maximizers: the exact maximum has \(Q=0\). The calculation instead shows how a flat direction can make imperfectly optimized solutions look very similar in likelihood. A careful check would tighten tolerances, evaluate the gradient, refine the optimization, and inspect the profile over the contrast. If the solutions then converge to the same maximum, the initial sensitivity was numerical; if estimates remain variable with negligible likelihood differences, the fitted likelihood is weakly informative along that combination at the available numerical and data resolution.

Curvature Is Not the Same as Sampling Variation

Observed curvature describes the shape of the likelihood for the data actually observed. Sampling variation describes how an estimator would vary across repeated samples from the assumed data-generating process. Under suitable regularity conditions, local likelihood curvature is related to the precision of likelihood-based estimation, but the two concepts are not interchangeable. In particular, a numerically small eigenvalue may reflect parameter units or poor numerical differentiation rather than substantial uncertainty in a meaningful scale.

A useful diagnostic sequence is to rescale parameters to interpretable units, recompute the Hessian with stable numerical methods, inspect eigenvectors as parameter combinations, and compare the local quadratic approximation with the relevant profile likelihood. Repeating the optimization from multiple starting values tests whether the reported solution is reproducible. These checks can identify weakly determined combinations in the fitted likelihood; they do not establish the model's identification assumptions or substitute for an analysis of sampling behavior.

Common Pitfall

A common mistake is to label the parameter associated with the smallest diagonal entry of the Hessian as the weakly identified parameter. This ignores off-diagonal terms: the weak direction can be a combination of parameters, as the \((1,-1)'\) eigenvector in the worked calculation demonstrates. The correction is to inspect eigenvalues together with their eigenvectors, account for parameter scaling, and use a profile likelihood when the inferential target is a particular parameter. Also, do not infer that a small eigenvalue proves nonidentification; it is evidence about local curvature in the observed likelihood, not a standalone identification test.

Key takeaway: Small Hessian eigenvalues, broad profile likelihoods, and sensitivity to starting values are related but distinct diagnostics. Interpret them together, check the parameter scale and numerical accuracy, and distinguish weak curvature in the observed likelihood from sampling variation across hypothetical repeated samples.

Check Your Understanding

Answer each question by referring to the likelihood geometry and the stated assumptions.

For a positive-definite observed information matrix, what do an eigenvector and its eigenvalue represent in the local quadratic approximation?
Why can a parameter combination have weak curvature even when each diagonal entry of the Hessian is relatively large?
In profiling a target parameter, why is the profile curvature generally different from the Hessian entry obtained while holding nuisance parameters fixed?
What checks would help distinguish optimizer sensitivity caused by a flat likelihood from sensitivity caused by numerical failure?
Why does a small observed-information eigenvalue not, by itself, prove that a model is unidentified?