Why a Failed Fit Needs More Than an Optimizer Message
A model fit can fail because the data do not contain enough information to estimate its parameters precisely. It can also fail because the implemented model is wrong, the data are misaligned with the records used in the likelihood, or numerical calculations are unreliable. These explanations can produce similar symptoms: large standard errors, parameters at bounds, poor convergence, or unstable estimates. The symptoms alone do not identify the cause.
A useful diagnosis combines three views of the same fit. Prediction checks ask whether the model produces the intended values for known inputs and whether those values are plausible against the observations. Likelihood contributions show which observations or individuals account for poor fit. Parameter profiles reveal how strongly the likelihood changes as a parameter moves away from its estimate. The conclusion should come from agreement among these checks, not from any one diagnostic in isolation.
The distinction is not always binary. A model can be correctly implemented but systematically inadequate for the observations; data can be sparse for one parameter but informative for another; and a numerical defect can be especially visible only in a poorly supported region of parameter space. The checks below are intended to locate evidence and narrow explanations, not to certify a model from a single plot or statistic.
Prediction Checks: Verify the Calculation Before Interpreting the Fit
A prediction check starts with a clearly specified input and a prediction that can be calculated independently or inspected for basic properties. For each record, verify the subject, dose, time, covariates, units, and prediction type. A population prediction and an individual prediction conditional on estimated random effects answer different questions; compare like with like. For a concentration model, predictions should also have concentration units, and any exponent must be dimensionless.
When checking predictions against observations, residuals such as observed minus predicted values can reveal systematic patterns, outliers, or a scale mismatch. The interpretation depends on the observation model: a residual of 1 mg/L is not equally surprising at every predicted concentration if the model assumes proportional rather than constant error. Prediction checks should therefore use the model’s stated error structure and should not treat every visible discrepancy as a coding bug.
Worked Example: An Independent Check of a Concentration Prediction
Suppose a hypothetical one-compartment intravenous bolus model has dose \(100\) mg, volume \(V=20\) L, and clearance \(CL=2\) L/h. For this model, the concentration at time \(t\) hours is
Assume the intended prediction is the population prediction, with no random-effect adjustment. The initial concentration is \(100/20=5\) mg/L, and the elimination rate \(CL/V=2/20=0.1\) h\(^{-1}\). At \(t=2\) h,
At \(t=8\) h,
Now suppose the implementation instead uses \(\exp(-CL\times t)\), omitting division by \(V\). It returns \(5\exp(-2\times2)=0.09158\) mg/L at 2 h and \(5\exp(-2\times8)=5.63\times10^{-7}\) mg/L at 8 h. This is evidence of a coding defect: \(CL\times t\) has units of litres, so it cannot be the dimensionless exponent required by the stated equation. An independent calculation identifies the discrepancy without relying on optimizer behavior. If observations are near the correctly calculated values, their disagreement with the implementation is not evidence that the data fail to support the model.
A useful prediction check asks whether the implementation reproduces simple limiting cases as well as ordinary examples. For the bolus model above, \(C(0)\) must equal Dose divided by \(V\), and concentration must decrease over time when \(CL\) and \(V\) are positive. A failure of such a check points toward implementation or input handling. Passing it does not establish that the model is biologically adequate or that the data identify all parameters.
Likelihood Contributions: Locate What Drives the Fit
For independent observations with density \(f(y_i\mid\theta)\), the log-likelihood is the sum of the observation-level log-likelihood contributions:
In a population model, the unit of contribution may instead be an individual, with that individual’s data jointly contributing to a marginal likelihood after accounting for random effects. The decomposition must match the likelihood actually fitted; observation-level terms from a conditional calculation are not automatically the same as marginal individual contributions.
A very low contribution can arise from a genuinely unusual observation under a correctly implemented model, from model misspecification, from an erroneous record or record-to-prediction match, or from a numerical calculation failure. It is a flag for inspection, not a diagnosis by itself. Check the source data, units, event history, prediction, and contribution calculation for flagged records. Then assess whether the remaining pattern is compatible with the assumed observation model.
Worked Example: A Single Record Dominates the Log-Likelihood
Consider four hypothetical independent observations with normal errors of known standard deviation \(\sigma=1\) concentration unit. The predictions are \(10,\ 12,\ 14,\ 16\) mg/L, and the observed values are \(10.2,\ 11.8,\ 14.1,\ 40.0\) mg/L. For each record,
The residuals \(y_i-\hat{y}_i\) are \(0.2,\ -0.2,\ 0.1,\ 24.0\) mg/L. Substituting \(\sigma=1\) mg/L gives contributions of approximately \(-0.9389,\ -0.9389,\ -0.9239,\) and \(-288.9189\), respectively. For example, the fourth contribution is
The total log-likelihood is approximately \(-291.7208\), and the fourth record dominates it. This warrants checking that the observation is \(40.0\) mg/L, that it belongs to the stated individual and time, and that its prediction and units are correctly matched. If those checks pass, the discrepancy may reflect a data point poorly described by the model; it does not, by itself, prove a software defect. If the recorded value was intended to be \(14.0\) mg/L, correcting that verified data error would change the residual to \(-2.0\) mg/L and the contribution to \(-2.9189\), a materially different result.
Parameter Profiles: Measure How Much the Data Constrain an Estimate
A parameter profile holds a parameter of interest at a sequence of fixed values and, at each value, maximizes the likelihood over the other parameters. If \(\psi\) is the parameter of interest and \(\lambda\) denotes nuisance parameters, the profile log-likelihood is
The profile is informative only if each point is evaluated reliably and the nuisance parameters have been adequately optimized. A broad, smooth profile with a shallow maximum can indicate weak information about \(\psi\). An irregular profile, abrupt jumps, or different values at the same \(\psi\) under apparently equivalent evaluations can instead indicate optimization or numerical trouble. Neither shape alone proves the cause; predictions and likelihood calculations at selected profile points should be checked.
Worked Example: A Broad Profile for a Mean Concentration
Suppose four hypothetical measurements, in mg/L, are \(9,\ 10,\ 11,\ 10\). For illustration, assume independent normal errors with known \(\sigma=2\) mg/L and an unknown common mean \(\mu\). The sample mean is \(\bar{x}=10\) mg/L, which maximizes the likelihood. The residual sum of squares at this estimate is
At a proposed mean \(\mu\), the residual sum of squares is \(2+4(\mu-10)^2\), with units \((\text{mg/L})^2\). The likelihood-ratio deviance relative to the maximum is therefore
where \(\mu-10\) is expressed numerically in mg/L. At \(\mu=10.5\) mg/L, \(D=0.25\), so this value is close to the maximum on the likelihood scale. Using the usual approximate 95% one-parameter likelihood-ratio cutoff of 3.84 gives \(D(\mu)\leq3.84\), or \(8.04\leq\mu\leq11.96\) mg/L. The wide interval relative to the estimate indicates limited precision from these four observations under the stated model. Because the example’s likelihood is calculated directly, that broad profile is not a sign of numerical failure. In more complex models, the same interpretation requires checking that the profile itself has been evaluated reliably.
Justification and Limits of the Profile Interpretation
The likelihood-ratio profile compares the best fit at a fixed \(\psi\) with the unrestricted best fit. Its deviance is \(D(\psi)=2\{\ell(\hat{\psi},\hat{\lambda})-\ell_{\mathrm{prof}}(\psi)\}\), which is nonnegative when the unrestricted maximum is found and both calculations are valid. Near a regular, well-defined interior maximum, a local quadratic approximation gives
This explains why a sharply curved profile corresponds to a small standard error and a flat one to a large standard error. The approximation and the familiar chi-square cutoff rely on regularity conditions, including an identifiable interior parameter and a sufficiently informative likelihood. They can be unreliable at boundaries, with weakly identified parameters, or when the likelihood is non-smooth. A profile interval should not be presented as an exact guarantee in those cases.
A Hessian-based standard error is a local summary of curvature near the estimate; a profile shows how the likelihood behaves over a wider range and can reveal asymmetry or flat regions. Neither replaces checking whether the likelihood calculation is correct. Similarly, a residual plot can show systematic prediction errors but cannot establish that the likelihood code is correct. Use the tools for complementary purposes.
A Practical Diagnostic Sequence
Specify the model, parameter values, prediction type, units, observation model, and contribution unit before interpreting a diagnostic.
Use simple cases, dimensional consistency, and known input records to test that the implementation follows the stated equations.
Identify unusually influential observations or individuals, then verify their records, predictions, units, and contribution calculations.
Look for a broad but smooth likelihood region versus irregular evaluation, and verify selected profile points rather than relying on the plot alone.
Describe whether evidence favors limited information, a data or model discrepancy, or an implementation/numerical defect, and state what remains unresolved.
Failure Modes and Common Mistakes
Several warning signs are especially useful when the three checks are considered together. Predictions that violate units or basic limiting behavior point toward incorrect equations or inputs. A single extreme contribution suggests checking that record carefully, but does not establish whether the cause is a data problem, model inadequacy, or implementation defect. A broad, smooth profile can be consistent with weak support; an irregular profile should prompt checks of likelihood evaluation and optimization at the affected parameter values.
Numerical error can imitate weak support by making nearby parameter values appear to have similar likelihoods, or imitate strong support through spurious jumps. Earlier material on numerical integration error, finite-difference gradient noise, nonfinite likelihood contributions, and sparse individual data provides relevant failure mechanisms. The practical response here is to test whether the predictions, contributions, and profile evaluations agree with the specified model and data.
- Mistake: “The optimizer converged, so the parameter is supported.” Convergence indicates that a stopping rule was met; it does not show that the likelihood is informative, the maximum is global, or the calculation is correct. Inspect the profile and verify predictions and contributions.
- Mistake: “One very low contribution proves a coding bug.” A valid observation can be improbable under the assumed model. Check record integrity and calculations first, then consider model-data discrepancy if both are correct.
- Mistake: “A flat profile proves the data are sparse.” A flat profile can also result from inaccurate or inconsistent evaluations. Verify selected profile points and their predictions before attributing the shape to limited support.
- Mistake: “A good prediction at one record validates the full model.” A single point checks only a narrow part of the calculation. Test multiple inputs, event times, and relevant parameter values, and inspect the likelihood terms used in fitting.
Check Your Understanding
Answer each question, then compare with the solutions.
- A model’s predictions are 3.0 mg/L while an independently calculated prediction at the same inputs is 4.5 mg/L. What should be checked before attributing the difference to weak data support?
- For independent normal observations with known standard deviation, what does one extremely low observation-level log-likelihood contribution establish, and what does it not establish?
- A parameter profile is broad and smooth, and independently checked likelihood values agree with the profile calculations. What is the most direct interpretation?
- Why can a Hessian-based standard error fail to reveal an asymmetric likelihood profile?
Solutions. (1) Verify the equation, parameterization, input records, units, prediction type, and implementation against the independent calculation; the discrepancy is evidence of a calculation mismatch, not weak support by itself. (2) It establishes that the observation contributes relatively little likelihood under the fitted model; it does not establish a coding bug, since a valid observation may be poorly predicted. Check the record and calculation, then assess model adequacy. (3) The data and model provide relatively weak precision for that parameter over the profiled range, subject to the specified model and likelihood assumptions. (4) The Hessian summarizes local curvature at the estimate. A profile explores a wider range, where curvature may change; a local quadratic approximation can therefore miss asymmetry.