Worked Diagnosis of an Unstable Likelihood Fit

Why diagnose the fit as a whole?

A selection-model likelihood can be numerically maximized even when the data provide little information about how dropout depends on an unobserved outcome. In that situation, the optimizer may report convergence, yet different starting values can lead to different solutions, the likelihood can be nearly flat along a curved direction, and a Wald standard error can be misleading. A useful diagnosis therefore combines evidence from several views of the same fitted likelihood: optimizer starts, contours, local curvature, and profiles.

The worked case below is hypothetical. It describes a longitudinal study in which every participant has a baseline outcome, but some participants drop out before the follow-up outcome is measured. Outcomes are expressed in standard-deviation units. The goal is not to decide whether the selection model is true; it is to assess whether its likelihood supports stable estimation of the dropout parameter.

Model and likelihood for the example

The bivariate normal distribution specifies the joint outcome density \(f(y_1,y_2)\). For a participant with observed follow-up, the likelihood contribution is the joint outcome density multiplied by the probability of observation. For a participant who dropped out, the follow-up value is unavailable, so its contribution integrates over all possible follow-up values:

The observed-data log likelihood is \(\ell(\theta)=\sum_i\log L_i\), where \(\theta\) collects the outcome and selection parameters. The integral must be evaluated numerically in this example; the diagnosis assumes that quadrature accuracy has already been checked. The results also assume that the optimizer is being applied to the intended likelihood, with parameters and outcome scaling implemented consistently. A failed check of either assumption can mimic statistical instability.

Compare starts before trusting a reported solution

A convergence message says that an algorithm met its stopping rule; it does not establish that it found the global maximum or that the parameters are well identified. Multiple starts should vary the initial selection parameters meaningfully, not just perturb the outcome parameters around one initial point. Compare the final log likelihoods and estimates, and inspect whether convergence was to an interior stationary point.

The absolute log-likelihood values are meaningful only for comparing fits to the same data under the same likelihood definition. Also, starts that converge to one solution do not prove that no other maximum exists. They provide evidence about the region explored, not a global guarantee.

Contours and local curvature reveal different features

A likelihood contour shows combinations of parameters with similar log likelihood. A narrow, elongated contour indicates that some combinations are much better determined than others. If increasing \(\gamma_2\) can be offset by changing the follow-up mean \(\mu_2\), for example, the likelihood may change very little along a diagonal path. This is a ridge: the parameter pair is weakly identified jointly, even if the likelihood has a well-defined maximum.

The observed information matrix at a fitted point is \(J(\hat{\theta})=-H(\hat{\theta})\), where \(H\) is the Hessian matrix of second derivatives of the log likelihood. Its eigenvectors give local directions in parameter space, and its eigenvalues give curvature in those directions. On a locally concave log likelihood at a local maximum, the Hessian is negative semidefinite and the observed information is positive semidefinite. Thus its eigenvalues are nonnegative; they are all positive only if the Hessian is negative definite at the fitted point. A zero eigenvalue indicates a locally flat direction, while a small positive eigenvalue indicates weak local curvature. A negative eigenvalue may signal that the point is not a local maximum, or that numerical derivative error needs investigation.

A contour plot and an eigenanalysis complement one another. The contour displays the shape over a chosen two-parameter slice; the Hessian summarizes curvature locally in the full parameter space. A reassuring contour in one slice cannot rule out a flat direction involving other parameters, and a local Hessian does not show whether the likelihood bends, levels off, or turns elsewhere.

Use a profile likelihood to examine the weak parameter

For a parameter of interest such as \(\gamma_2\), the profile log likelihood at a fixed value \(g\) is the largest log likelihood achievable by varying all nuisance parameters while setting \(\gamma_2=g\). This reoptimization matters: simply holding nuisance estimates fixed can exaggerate how much the likelihood changes. A nominal 95% likelihood-ratio profile interval includes values satisfying \(2\{\ell(\hat{\theta})-\ell_{\mathrm{prof}}(g)\}\leq 3.841\), using the one-degree-of-freedom chi-square cutoff. This calibration is an approximation and needs regularity conditions; if those conditions are doubtful, calibration by an appropriate parametric bootstrap is a possible alternative.

Bring the diagnostics together

For the hypothetical fit, the best starts give almost identical estimates and log likelihoods, while one start finds a substantially inferior solution. That calls for checking the competing solution, but it does not by itself invalidate the best fit. The small positive information eigenvalue and the elongated \((\mu_2,\gamma_2)\) contour indicate a weakly curved trade-off near that fit. The broad, asymmetric profile interval then shows that substantial changes in \(\gamma_2\) can be accommodated by reoptimizing other parameters.

These findings are consistent with a likelihood that has a high-likelihood region but weak information about the dropout association. They do not establish which mechanism produced the instability. Possible contributors include weak identification from the observed data, a nearby competing maximum, poor parameter scaling, inaccurate numerical integration, or derivative error. The checks should be targeted accordingly: verify the likelihood and quadrature, compare starts and optimization settings, inspect contours, and check local curvature and profiles. Do not interpret convergence alone as evidence that all parameters are well determined.

A Wald interval relies on a local quadratic approximation and the inverse observed information. Near a ridge, that approximation can be unrepresentative over the parameter range of interest, even when the optimizer reports a maximum. The profile interval instead allows nuisance parameters to adjust at each fixed value of the parameter of interest. Neither approach removes weak information from the data. Profile methods can also be unreliable if optimization fails at fixed parameter values or if the likelihood-ratio calibration is poor. A parametric bootstrap can assess calibration or sampling behavior under a specified fitted model, but it cannot make an uninformative likelihood informative.

Common pitfall and practical limits

A diagnosis is conditional on the model and computational implementation. The bivariate normal outcome assumption, logistic dropout model, independence across participants, and numerical integration all matter. If the outcome distribution or selection mechanism is misspecified, a stable maximum can still answer the wrong model-based question. Conversely, an unstable selection parameter need not make every estimand unusable: report which quantities are weakly supported, and avoid presenting a precise interpretation of an uncertain dropout mechanism. The model remains a sensitivity framework, not a guarantee that missing outcomes have been recovered from the observed data.