Why Record Optimization Diagnostics?
A GLMM fit is the result of a numerical search, not just a list of coefficient estimates. A report that gives only the estimates and says “converged” hides choices that can affect the result: the optimizer, its control settings, the starting values, warnings, and how the final point was checked. A reproducible record makes those choices visible and allows a reviewer to assess whether another fit of the same model reached a comparable solution.
The purpose is not to collect every available software message. It is to retain enough information to identify what was optimized, how it was optimized, what the computation reported, and whether independent numerical checks support the reported solution. A record can document an unsuccessful or questionable fit just as usefully as a successful one.
What a Record Needs to Specify
For orientation, consider a binary-outcome random-intercept GLMM. For individual \(i\) in group \(j\), let \(Y_{ij}\) be the binary response, \(x_{ij}\) a predictor, and \(b_j\) a group-specific random intercept. One possible model is
This statement identifies the conditional distribution and linear predictor, but it does not completely specify a numerical fit. A record should also say how the marginal likelihood was evaluated or approximated, which objective was minimized or maximized, and how the variance parameter was represented during optimization. For example, a software implementation might optimize a log standard deviation, \(\lambda=\log(\sigma_b)\), to keep \(\sigma_b\) positive. That parameterization matters when interpreting gradients and Hessians.
Record the following, using software-specific names where necessary:
- Model and data: response, fixed- and random-effects structure, analysis sample, missing-data handling, grouping variable, and any relevant data exclusions.
- Objective and parameterization: likelihood or approximation being optimized, whether the reported objective is a negative log likelihood, deviance, or another quantity, and the scale on which variance parameters were optimized.
- Optimization settings: optimizer name, tolerances, iteration or evaluation limits, and any other control settings that can affect termination or the numerical path.
- Initialization and messages: starting values, termination status, warnings, and whether the run stopped because of a tolerance, a limit, or another condition.
- Diagnostics and estimates: gradient information, Hessian or curvature checks, parameter estimates, and any boundary or singular-fit indication reported by the software.
- Provenance: software and relevant package versions, code or model specification, and a version or identifier for the analyzed data.
A tolerance should be reported with its name and meaning, not just as a bare number. A gradient tolerance, a relative change in objective tolerance, and a parameter-step tolerance are not interchangeable. Nor is a gradient norm comparable across implementations if the objective scaling, parameter scaling, or gradient definition differs. The optimization settings therefore belong beside the software and objective specification.
Why Gradients and Curvature Belong in the Record
Suppose the software minimizes a differentiable objective \(Q(\theta)\), where \(\theta\) is the vector of parameters in the optimizer's coordinates. At an interior local minimum, a necessary first-order condition is that the gradient is zero. A second-order necessary condition is that the Hessian is positive semidefinite. For a well-curved strict local minimum, the Hessian is positive definite. These are local checks: they do not prove that the model is appropriate, that the global minimum has been found, or that the likelihood approximation is accurate.
The gradient describes the local slope. A small gradient is evidence that further small moves may not reduce the objective much, but “small” depends on scale. The Hessian describes local curvature. A negative eigenvalue of the Hessian for a minimization objective indicates a direction of negative curvature, inconsistent with an interior local minimum. A very small positive eigenvalue indicates a weakly curved direction, where numerical uncertainty can be substantial.
The justification is the local Taylor expansion. For a small change \(h\) in the parameter vector,
If the gradient is not close to zero, the linear term can still suggest a direction of improvement. If the Hessian has a negative-curvature direction, the quadratic term can indicate a local decrease away from the purported minimum. If curvature is nearly zero in a direction, estimates along that direction can be sensitive to small changes in data or computation. This is why recording only a termination message is not enough.
At a constrained boundary, the interior condition \(\nabla Q(\hat{\theta})=0\) need not hold in every raw coordinate. The relevant condition depends on the constraints and parameterization. Record the parameter scale and boundary status; do not label a nonzero raw gradient at a boundary as automatic proof of failure. Likewise, software can report a Hessian calculated in a transformed coordinate system. Its result must be interpreted on that scale.
Worked Example: Recording an Interior Random-Intercept Fit
Suppose a hypothetical binary-outcome data set contains 8 clinics and 20 patients per clinic. A treatment indicator \(x_{ij}\) equals 1 for treated patients and 0 otherwise. The stated model is the random-intercept model above. The following numbers are illustrative optimizer output for a fit using an explicitly recorded likelihood approximation and the objective \(Q\) equal to the negative approximate log likelihood.
The record gives optimizer “Optimizer A,” starting values \((\beta_0,\beta_1,\lambda) = (0,0,\log(0.5))\), and the declared stopping settings: gradient tolerance \(0.002\), maximum 1,000 iterations, and maximum 2,000 objective evaluations. The termination message says the gradient criterion was met after 84 iterations. No warning was issued.
The final estimates are \(\hat{\beta}_0=-1.20\), \(\hat{\beta}_1=0.55\), and \(\hat{\lambda}=\log(0.60)=-0.5108\), so \(\hat{\sigma}_b=\exp(-0.5108)\approx0.600\). The gradient in the recorded optimizer coordinates is \((0.0008,-0.0011,0.0004)\). Its maximum absolute component is
For this declared componentwise criterion, the gradient satisfies the stated tolerance. Suppose the reported Hessian for \(Q\) in these same coordinates is diagonal with entries \(18.0,\ 7.50,\ 4.20\). Its eigenvalues are those diagonal entries, all positive, so this illustrative Hessian is positive definite. These checks support an interior local minimum; they do not by themselves establish global optimality or validate the model.
For context, the conditional treatment odds ratio is \(\exp(\hat{\beta}_1)=\exp(0.55)\approx1.73\). This estimate is conditional on the random intercept under the stated model. The record should preserve the coefficient and its scale as well as any transformed summary; reporting only “odds ratio 1.73” would omit the fitted parameterization and model context.
Comparing Fits Fairly
Comparisons are informative only when the fits refer to the same data, model, objective definition, likelihood approximation, and parameterization. If any of these change, the comparison is no longer simply a check that different numerical runs found the same solution. For instance, a different analysis sample or approximation can change the objective and estimates even if the optimizer is flawless.
When those elements are held fixed, compare at least the objective value, parameter estimates on a common scale, termination information, gradient diagnostics, and Hessian checks. Small differences across runs can reflect numerical precision. Larger differences may indicate that runs reached different regions, that one run terminated early, or that the objective is weakly curved. A smaller objective value alone is not sufficient evidence: the associated point should also have appropriate local diagnostics and a valid termination status.
Worked Example: Comparing Two Optimizers on the Same Fit
Suppose the same hypothetical data and GLMM are fit twice, with the same objective and parameterization. Run A uses the settings and estimates in the preceding example. Run B uses “Optimizer B,” the same starting values, and documented tolerances. It reports \(\hat{\beta}_0=-1.199\), \(\hat{\beta}_1=0.552\), \(\hat{\sigma}_b=0.598\), and \(Q=142.3810\). Run A reports \(Q=142.3812\). Both runs issue no warnings, satisfy their recorded gradient criteria, and have positive-definite Hessian checks.
The objective difference is
The absolute differences in the common-scale estimates are \(0.001\) for the intercept, \(0.002\) for the treatment coefficient, and \(0.002\) for the random-intercept standard deviation. For example,
These close results are evidence that the two recorded runs found numerically similar solutions for this specification. They do not establish that the model is correct. The record should retain both optimizer names, settings, starts, messages, diagnostics, and estimates; replacing the details with “the fits agreed” would make the comparison difficult to audit.
Warnings, Boundaries, and Interpretive Limits
Save warnings exactly or summarize them faithfully, even if the software also returns estimates. A warning about a gradient, Hessian, iteration limit, or boundary condition is part of the fit result. Suppressing a warning in a final report makes it harder to distinguish a clean run from one that produced numbers despite a computational concern. A warning is not itself a diagnosis of the cause; it should be read alongside the numerical evidence.
Boundary estimates require explicit documentation. For a variance component, a fitted value at or near zero can be a legitimate boundary result, but it changes how routine interior diagnostics should be interpreted. Record the estimate, its scale, any boundary or singular-fit message, and the gradient and curvature information as reported. Do not turn an approximate zero into a positive value by rounding without noting the unrounded estimate.
Worked Example: Reporting a Near-Boundary Estimate
In a hypothetical fit of the same random-intercept model, suppose the optimizer reports \(\hat{\sigma}_b=0.004\), with no iteration-limit warning, and gives \(\hat{\lambda}=\log(0.004)\approx-5.521\). The software also reports a boundary or singular-fit indication. The correct record preserves both the estimate \(0.004\) and the indication; it should not simply state that the random-intercept variance was “zero” if the unrounded estimate is positive.
On the variance scale, the estimate is
This small estimated variance and the boundary indication are relevant to interpretation, but they do not alone prove that the optimizer failed. The record should include the gradient and Hessian information in the optimizer's coordinates and state any limitation the software reports for those checks. Since \(\lambda\) is far below zero, curvature on the log-standard-deviation scale may not answer the same question as curvature on the variance scale. Avoid presenting a routine interior Hessian check as conclusive evidence of a regular interior solution.
Common Pitfalls and Alternatives
| Incomplete or misleading practice | More defensible record |
|---|---|
| “The model converged,” with no supporting detail | Give the optimizer, its settings, termination message, warnings, and numerical checks. |
| Reporting a gradient norm without defining its scale | Identify the gradient or norm, parameter coordinates, objective scaling, and tolerance used. |
| Comparing objective values from different objectives or samples | First verify that data, model, objective, and approximation are identical; otherwise describe the difference explicitly. |
| Dropping a warning because estimates were returned | Retain the warning and report the estimates as output from a run with that warning. |
| Assuming a positive-definite Hessian proves model adequacy | Interpret it only as local curvature evidence for the stated objective and parameterization. |
A convergence flag is a convenient summary, but it is not a substitute for the diagnostic record. Re-running the same optimizer from different starting values can reveal sensitivity to initialization. Running a second optimizer can provide a useful numerical comparison. These are complementary checks, not replacements for reporting the settings and diagnostics of each run. Profile likelihood or other inferential checks address different questions and should not be presented as though they were optimizer logs.
A well-documented fit can still be limited by a weakly identified parameter, sparse data, a fragile likelihood approximation, or a poorly chosen model. Diagnostics describe computational behavior; they do not establish that the data support every parameter or that the model assumptions are scientifically appropriate. Conversely, a concerning diagnostic does not by itself identify the remedy. The record makes the evidence reviewable without confusing documentation with a full investigation of the cause.
A Compact Reproducible Record
A practical report can use a table or a structured text file, provided each item can be linked to a particular fit. Include enough precision to distinguish runs and avoid rounding away small but relevant differences. Preserve full software output where feasible, while summarizing the essential details in the report.
Record the data version, analysis sample, model formula, likelihood approximation, and objective definition.
Record optimizer, parameterization, tolerances, limits, and starting values.
Save termination status, warnings, estimates, gradient information, Hessian checks, and boundary indications.
For every additional run, record the same fields and state which model and data features were held fixed.
This structure keeps the computational record reproducible and the interpretation appropriately limited. It also helps readers separate three questions: what model was fit, what the optimizer reported, and what evidence supports treating the numerical result as stable enough for subsequent analysis.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- A report gives estimates and the word “converged,” but no optimizer or tolerance. Name two important omissions and explain why they matter.
- For a minimization objective, a recorded Hessian has a negative eigenvalue at an alleged interior solution. What does this indicate, and what does it not establish?
- Two runs of the same model report objectives 96.250 and 96.248. What additional information is needed before treating them as comparable?
- A variance component is estimated as \(0.003\), and the software reports a boundary indication. What should the fit record preserve?
Solutions. (1) The optimizer and its tolerance settings are two omissions. The optimizer identifies the numerical procedure; tolerances help explain the termination rule, and both are needed to reproduce and assess the run. (2) A negative Hessian eigenvalue indicates negative curvature in some direction, inconsistent with an interior local minimum under the stated objective. It does not by itself identify the cause or prove the model is inappropriate. (3) Confirm the same data, model, objective definition, approximation, and parameterization; then compare estimates, warnings, termination information, gradients, and Hessian checks. (4) Preserve the unrounded estimate, its parameter scale, the boundary indication, warnings, termination details, and gradient and Hessian information with their parameterization. Do not silently replace the estimate by zero or treat an interior Hessian check as conclusive.