Flat Likelihoods And Numerical Identification Diagnostics

Why Flat Likelihood Directions Need Diagnosis

A nonlinear likelihood optimizer can report convergence even when the fitted parameters are imprecise or when the numerical problem is poorly scaled. These are different issues. A weakly determined parameter combination means the data and model yield similar likelihood values over a substantial range in some direction through parameter space. Numerical ill-conditioning means the computation has difficulty representing or optimizing the problem, often because parameters use very different scales. Rescaling may help with the latter, but it does not create information that the data do not contain.

The local Hessian, profile likelihoods, and sensitivity to starting values provide complementary diagnostics. None is decisive in isolation. A small Hessian eigenvalue may be caused by units; a profile can reveal curvature that a local quadratic approximation misses; and different numerical estimates may reflect either a flat ridge or an optimizer that has not reached a satisfactory solution.

Curvature and Hessian Eigenvalues

Let \(\ell(\theta)\) be the log-likelihood for a parameter vector \(\theta\), and suppose an interior estimate \(\hat{\theta}\) is a local maximum. The observed information is the negative Hessian of the log-likelihood evaluated at that estimate. At an interior local maximum, the Hessian is negative semidefinite, so the observed information is positive semidefinite. Strictness alone does not imply positive definiteness, because a strict maximum can have zero curvature in one or more directions.

Definition: The observed information matrix at \(\hat{\theta}\) is \(J(\hat{\theta})=-\nabla^2\ell(\hat{\theta})\). Its eigenvectors give directions in parameter space, and the corresponding eigenvalues measure local curvature in those directions. A small positive eigenvalue indicates low curvature along its eigenvector, so relatively large movement in that direction produces only a small local change in log-likelihood. An eigenvalue that is zero or nearly zero calls for further investigation; it does not, by itself, establish why the direction is flat.

If an eigenvalue is negative at a purported interior maximum, the Hessian is not negative definite for the log-likelihood. The point may be a saddle point, a minimum in at least one direction, or a point at which optimization has not settled. At a constrained or boundary solution, ordinary interior curvature conditions need not apply in the same way, so the feasible directions also matter.

Eigenvalues depend on the parameter coordinates and their units. Measuring one parameter in dollars rather than thousands of dollars can make the raw eigenvalues look very different. The ratio of largest to smallest eigenvalue, called the condition number for a positive-definite matrix, also changes under rescaling. Thus raw eigenvalues and condition numbers are most interpretable after parameters have been put on meaningful, comparable scales. A broad, nearly level direction that remains after sensible scaling is stronger evidence of weak determination than a small raw eigenvalue alone.

Worked Example: Finding a Flat Combination from a Quadratic Likelihood

Suppose a hypothetical log-likelihood for two parameters \(\beta\) and \(\gamma\), near its maximum, is exactly

\ell(\beta,\gamma)=C-\frac{1}{2}\left(4\beta^2+6.25\beta\gamma+2.5\gamma^2\right),

where \(C\) is a constant. The gradient is zero at \((\beta,\gamma)=(0,0)\). The observed information there is

J= \begin{pmatrix} 4 & 3.125\\ 3.125 & 2.5 \end{pmatrix}.

Its trace is \(6.5\), and its determinant is \(4(2.5)-3.125^2=0.234375\). Therefore its eigenvalues solve \(\lambda^2-6.5\lambda+0.234375=0\), giving approximately \(6.4637\) and \(0.03626\). Both are positive, consistent with a strict local maximum, but the smaller eigenvalue indicates much less curvature in one direction than in the other. The associated weak direction is approximately proportional to \((1,-1.27)\).

To see the same feature through a profile, hold \(\beta\) fixed and maximize over \(\gamma\). The derivative with respect to \(\gamma\) is

\frac{\partial \ell}{\partial \gamma}=-3.125\beta-2.5\gamma.

Setting this derivative to zero gives the maximizing nuisance value \(\hat{\gamma}(\beta)=-1.25\beta\), since the second derivative with respect to \(\gamma\) is \(-2.5<0\). Substitution gives

\ell_p(\beta)=\ell(\beta,-1.25\beta) =C-\frac{1}{2}(0.09375)\beta^2.

For example, at \(\beta=2\), the profiled value is \(C-0.1875\), while at \(\beta=0\) it is \(C\). The profile’s small curvature shows that \(\beta\) can move appreciably if \(\gamma\) moves with it, with little likelihood loss. It is the combination of parameters—not necessarily either parameter considered while the other is fixed—that is weakly determined.

Profile Likelihoods Reveal Joint Movement

For a scalar parameter of interest \(\psi\), write the remaining parameters as \(\lambda\). The profile log-likelihood is the best log-likelihood available at each fixed value of \(\psi\):

Definition: The profile log-likelihood is \(\ell_p(\psi)=\sup_{\lambda}\ell(\psi,\lambda)\), where the supremum is taken over the allowed values of the nuisance parameters. Profiling allows nuisance parameters to adjust as \(\psi\) changes; it is not the same as holding them fixed.

A profile is useful because it displays the likelihood along a parameter direction after accounting for the best compensating movement of the other parameters. A broad profile indicates that a range of values for \(\psi\) can be accompanied by nuisance-parameter values that fit nearly as well. The corresponding uncertainty may be highly asymmetric, so a single local standard error can give an incomplete picture. A profile may also reveal multiple peaks or a one-sided plateau that a Hessian evaluated at just one point cannot show.

The local relationship between a profile and the Hessian can be derived for a smooth likelihood with a negative-definite Hessian at an interior maximum. Partition the observed information into blocks for \(\psi\) and \(\lambda\):

J= \begin{pmatrix} J_{\psi\psi} & J_{\psi\lambda}\\ J_{\lambda\psi} & J_{\lambda\lambda} \end{pmatrix}.

Near the maximum, the log-likelihood change is approximately minus one half of the corresponding quadratic form in parameter changes. When \(\psi\) is displaced, maximizing that quadratic approximation over nuisance changes gives \(\Delta\lambda=-J_{\lambda\lambda}^{-1}J_{\lambda\psi}\Delta\psi\). Substitution leaves the profile curvature

J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi}.

This quantity is the Schur complement of \(J_{\lambda\lambda}\) in \(J\). It is no larger than the curvature \(J_{\psi\psi}\) obtained by holding nuisance parameters fixed, when the blocks satisfy the stated positive-definiteness conditions. Allowing nuisance parameters to adjust can therefore expose weaker effective curvature than a conditional slice suggests. This local derivation does not imply that a profile is globally quadratic; the actual profile should be examined over a meaningful range.

Separating Scale Problems from Weak Determination

A practical check for scale-induced ill-conditioning is to re-express parameters on scientifically meaningful scales and recompute the optimization and curvature diagnostics. If the apparent extreme condition number largely disappears after scaling, while the likelihood is well curved over substantively relevant changes in each scaled parameter, the original warning may have been mostly numerical. If a broad likelihood ridge persists in the rescaled coordinates, it reflects limited information about a parameter combination rather than merely inconvenient units.

Worked Example: A Small Eigenvalue Caused by Units

Consider a hypothetical log-likelihood

\ell(a,b)=C-\frac{1}{2}\left[(a-2)^2+\left(\frac{b-2000}{1000}\right)^2\right],

with \(a\) measured on a unit scale and \(b\) measured in units for which changes of about 1,000 are relevant. The maximum is \((\hat a,\hat b)=(2,2000)\). In the original coordinates, the observed information is diagonal:

J_{(a,b)}= \begin{pmatrix} 1 & 0\\ 0 & 0.000001 \end{pmatrix}.

The raw condition number is \(1/0.000001=1{,}000{,}000\), which looks severe. Now define \(u=a-2\) and \(v=(b-2000)/1000\). In these coordinates, \(\ell=C-\tfrac12(u^2+v^2)\) and the information matrix is the identity matrix, with both eigenvalues equal to 1. The maximum is equally curved in these scaled directions. The small original-coordinate eigenvalue arose from measuring \(b\) in units 1,000 times larger than the scale of its relevant changes; it did not imply that the scaled parameter combination was weakly determined.

This conclusion relies on the likelihood actually having the stated curvature over meaningful changes. Rescaling cannot make a genuinely broad profile narrow. If, for example, many values of \(v\) still give nearly the same likelihood after this transformation, the data remain weakly informative in that direction.

Starting-Value Sensitivity as a Diagnostic

For a well-behaved likelihood with a unique, well-separated maximum, different reasonable starting values should ordinarily lead a reliable optimizer to the same solution, up to numerical tolerance. When estimates differ, compare more than the parameter values: check the attained log-likelihood, the gradient, the Hessian, constraints, and the parameters’ scale. Similar likelihood values with estimates spread along a direction of low curvature are consistent with a flat likelihood. Different likelihood values, a substantial gradient, or an indefinite Hessian instead suggest that at least some runs may not have found the same satisfactory maximum.

A convergence flag is an algorithmic stopping report, not a statistical identification result. In particular, a small gradient does not guarantee a precisely located estimate when curvature is also very small: in a weak direction, a modest gradient can correspond to a large movement in parameter space. Repeat optimization from dispersed starts, use suitable scaling and tolerances, and inspect profiles or likelihood contours before interpreting differing estimates as evidence about the underlying model.

Worked Example: Interpreting Nearby Fits Along a Flat Direction

Return to the quadratic log-likelihood in the first example. Suppose an optimizer stopped at \((\beta,\gamma)=(0.60,-0.75)\). The value of \(\gamma\) is the profile maximizer for \(\beta=0.60\), because \(-1.25(0.60)=-0.75\). The log-likelihood loss relative to the maximum is

\ell(0,0)-\ell(0.60,-0.75) =\frac{1}{2}(0.09375)(0.60^2) =0.016875.

The gradient at this point is \((-0.05625,0)\): the nuisance-parameter component is zero, and the remaining component is small. At \((-0.60,0.75)\), the log-likelihood loss is also \(0.016875\), and the gradient is \((0.05625,0)\). Thus two stopping points can have similar likelihood values while differing in both parameters. They lie along the weakly curved profile direction.

These points are not distinct exact maxima: the stated quadratic likelihood has its unique maximum at \((0,0)\). Rather, the calculation shows why finite stopping tolerances or limited numerical precision can leave noticeably different provisional estimates along a flat direction. A sound diagnosis is to verify that tighter tolerances and improved scaling move the fits toward the same maximum, while also reporting that the likelihood itself is weakly informative about movement along the profile. Reporting one provisional point as precisely determined would overstate what the likelihood supports.

A Practical Diagnostic Sequence

Verify the candidate solution.
Check the log-likelihood, gradient, constraints, and Hessian at the reported estimate. An indefinite observed information matrix at an interior candidate warrants further optimization or investigation.

Inspect curvature on sensible scales.
Rescale or reparameterize quantities with very different units, then assess eigenvalues and conditioning in the transformed coordinates. Interpret directions as parameter combinations, not just isolated coefficients.

Profile suspicious parameters.
Fix a parameter at a grid of values and maximize over the nuisance parameters at each value. Record the log-likelihood and whether each nuisance optimization succeeded.

Repeat from dispersed starts.
Compare estimates and attained likelihoods, and re-run with better scaling or stricter tolerances. Similar likelihoods along a persistent broad profile suggest weak determination; unstable, lower-likelihood, or poorly converged runs suggest numerical trouble that must be resolved.

Common Pitfall and Reporting

A common mistake is to see one tiny raw Hessian eigenvalue and conclude that the model is unidentified. That reasoning ignores parameter units and the local nature of Hessian curvature. The correction is to check the eigenvector, rescale parameters meaningfully, and inspect a profile or other likelihood display. Conversely, a well-conditioned Hessian at one point does not rule out distant peaks, asymmetry, or a broad region outside the local approximation. Diagnostics should be interpreted together.

When a weak direction remains, describe which parameter combination moves, how the likelihood changes over the range examined, and whether the result was stable across starts and scaling choices. Distinguish a numerical failure from a successfully optimized but weakly informative likelihood. This distinction matters because numerical remedies can improve computation, whereas a genuine lack of curvature is a limitation of what the likelihood determines from the available model and data.

Key takeaway: Hessian eigenvalues describe local curvature, profiles describe likelihood loss after nuisance parameters adjust, and starting-value sensitivity tests optimization stability. Scale checks help separate numerical ill-conditioning from genuinely weakly determined parameter combinations.

Check Your Understanding

Use the diagnostics and definitions above to answer these questions.

What does a small positive eigenvalue of the observed information indicate locally, and why should its units be considered?
In a profile likelihood, what happens to the nuisance parameters as the parameter of interest is varied?
Why can the profile curvature for one parameter be smaller than the curvature found by holding nuisance parameters fixed?
Suppose rescaling makes the information matrix well-conditioned, but the profile remains broad over meaningful changes. What is the appropriate interpretation?
Two starting values produce different estimates with nearly identical log-likelihoods. Name two further checks that help determine whether this reflects a flat direction or unsuccessful optimization.