Why Compare Objective Functions?
When a submitted pharmacometric analysis is rerun, the reported objective function (OFV) provides a compact check that the run behaves as expected. A close match can support reproducibility; a noticeable difference can signal a change in the input data, model implementation, estimation settings, or optimization outcome. But the OFV is a summary, not an explanation. A difference by itself does not identify which part of the analysis changed.
The useful question is therefore not simply “Are the two OFVs equal?” It is “Were the runs intended to evaluate the same objective under comparable conditions, and, if so, is their difference within an acceptable numerical tolerance?” Answering this requires an explicit comparison convention and a controlled review of the run inputs.
Conditions for a Meaningful Run-to-Run Comparison
A direct OFV comparison is most interpretable when both runs evaluate the same statistical objective on the same observations and reach comparable solutions. In practice, this means checking more than the model file’s name. Confirm that the runs use the same analysis records, dose and observation data, handling of missing or censored values, likelihood contributions, model code, fixed and estimated parameters, and objective-function definition. Also compare the estimation method, numerical settings, software version, and any random seed or other source of stochastic variation.
Some differences in the reported value can arise even when analysts intend the runs to match. For example, a changed record-selection rule can alter which observations contribute to the likelihood. A code edit can change a residual-error expression or a covariate relationship. Different integration or optimization settings can change numerical approximations or which solution the optimizer reaches. In each case, the difference may be real and reproducible, but it is not evidence of a mere repeatability discrepancy.
When all inputs and settings are identical, a deterministic estimation procedure should ordinarily return the same result up to the effects of numerical precision and optimization tolerances. Exact equality is not a universal requirement: software versions, numerical libraries, machine architecture, starting values, and termination criteria can produce small differences. Conversely, there is no universal OFV threshold that separates a harmless discrepancy from a material one. The appropriate tolerance should be justified for the software, model, and review purpose, rather than chosen after seeing the result.
Worked Example: A Close Replication
Suppose a submitted run reports an OFV of 1452.381 and a replicated run reports 1452.409. The reviewer has confirmed that both runs use the same data records, model code, likelihood, fixed parameters, estimation method, and numerical settings. Both runs also report successful termination.
Using the stated sign convention:
The replicated OFV is 0.028 units higher. If the review’s documented tolerance for this exact run setup is 0.05 OFV units, then the difference is within that tolerance. The appropriate conclusion is that the two reported objective-function values agree within the specified tolerance—not that they are mathematically identical or that every aspect of the runs has been proven equivalent. The input and output checks still matter.
If no tolerance was specified in advance, the reviewer should not declare 0.028 acceptable merely because it looks small. The next step is to compare the run records and determine a defensible tolerance, or to rerun both analyses under a common controlled setup.
What the Difference Can—and Cannot—Tell You
An OFV difference is a useful discrepancy flag, but it does not decompose into contributions from particular records, model components, or settings. A lower value in one run cannot by itself establish that the model is better, that the code change was correct, or that a dataset alteration improved the analysis. Those interpretations require understanding what changed and whether the compared objective values are on the same basis.
For example, likelihoods calculated on different observation sets generally represent different data contributions. Even if the same model code is used, comparing the resulting OFVs as if they were two optimization results on one fixed dataset can be misleading. Likewise, if one run adds a penalty or uses a different likelihood convention, the numerical difference may not measure a change in fit on a common scale.
A likelihood-ratio comparison is a specific, conditional use of an OFV difference. If model A is nested within model B, both models are fitted to the same data using comparable likelihoods, and the usual regularity conditions hold, then:
Under the null hypothesis that the restrictions defining model A are correct, this difference has an approximate chi-square distribution with degrees of freedom equal to the number of independent restrictions, under standard likelihood-ratio assumptions. This result does not make every run-to-run difference a model-comparison statistic. In particular, it is not justified when the data differ, the objective functions are not comparable, or the models and hypotheses do not meet the required conditions. For a replication review, the main task is to diagnose comparability, not to attach a likelihood-ratio p-value to an unexplained discrepancy.
Worked Example: Different Observation Records
A submitted run reports an OFV of 602.7 using 240 observations. A replicated run reports 594.9, but the reviewer discovers that two observations were excluded during replication. The same model code and estimation method were used.
The arithmetic difference is:
The replicated value is 7.8 units lower. However, the likelihood contributions do not come from the same observation set, so this result is not a controlled check that the submitted run was reproduced. It also does not show that excluding the observations improved the model. The reviewer should trace the record-selection difference, establish whether the two exclusions were intended, and rerun both analyses on the same specified records before interpreting an OFV difference as a replication result.
A Controlled Review Strategy
When the OFVs differ, use a staged comparison that makes one source of change testable at a time. First verify that the compared values use the same objective-function definition and sign convention. Then check the data, model implementation, and estimation controls against the submitted run record. If a difference is found, correct or document it before drawing a conclusion about reproducibility.
Record the two OFVs, their software-defined meaning, the sign convention for \( \Delta_{\mathrm{OFV}} \), and the tolerance that will be used to judge agreement.
Compare the input dataset and the included analysis records, including doses, observations, exclusions, and any transformations or derived variables relevant to the likelihood.
Compare the executable model code and its fixed parameters, then check estimation method, numerical controls, software version, starting values, and any stochastic settings.
Where feasible, rerun with the submitted data, code, and settings held fixed. If a suspected change must be tested, change that factor alone and retain the run record for each comparison.
Report whether the values agree within the justified tolerance, and identify the demonstrated source of any material difference. If the source is unresolved, report it as unresolved rather than inferring it from the sign or size of the OFV difference.
Worked Comparisons for Code and Settings
Worked Example: A Model-Code Change
Two runs use the same 180 observations and estimation controls. The submitted model reports an OFV of 930.6. The replicated model reports 927.9. A code review finds that the replicated version contains a changed residual-error expression.
The difference is:
The replicated value is 2.7 units lower, but the runs do not evaluate the same model. The scalar difference cannot establish whether the code change is intended or correct. The reviewer should compare the implemented expression with the written specification, determine whether the change was authorized, and, if the purpose is replication, rerun the submitted code. If the change is intentional, describe the two values as results from different model implementations rather than as duplicate runs.
Worked Example: A Change in Estimation Settings
A submitted run reports an OFV of 871.4. A replicated run reports 869.1. The data and model code match, but the replicated run uses a different numerical integration setting.
The signed difference is:
This difference is associated with runs that used different estimation settings. It is not yet possible to say whether it reflects a change in numerical approximation, a different optimization outcome, or both. To isolate the setting, run the submitted configuration again under the submitted settings and then run the alternative setting with all other inputs held fixed. Record whether each run terminates successfully and whether the OFV is reproducible within repeated runs under its own configuration. Do not claim that the lower reported OFV alone proves the alternative setting is preferable.
Diagnostics, Alternatives, and Limitations
A useful review distinguishes a numerical discrepancy from a difference in the analysis being evaluated. Warning signs include inconsistent record counts, unexpected exclusions, changed model files, mismatched fixed parameters, different objective-function definitions, nonmatching software or estimation settings, and inconsistent termination outcomes. A large OFV discrepancy with no documented input change calls for investigation; it is not a diagnosis by itself. A small discrepancy also does not rule out a data or code difference that happens to have little effect on the total value.
The main alternatives serve different purposes. A direct OFV comparison is efficient for checking a controlled replication, but it is only a summary. A controlled rerun is more informative for isolating a suspected source because it holds other factors fixed. A likelihood-ratio comparison addresses a specific nested-model question under its assumptions; it is not a general substitute for tracing run differences. Reviewing records, code, and settings explains the provenance of a discrepancy, which the OFV alone cannot do.
| Approach | Best use | Main limitation |
|---|---|---|
| Direct OFV comparison | Quick check of two runs intended to be identical | Does not identify the cause of a difference |
| Controlled rerun | Testing whether one changed factor accounts for a discrepancy | Requires a reliable record of inputs and settings |
| Likelihood-ratio comparison | Testing a specified nested-model restriction under standard conditions | Not valid as a generic interpretation of different run values |
| Data, code, and settings audit | Tracing the source and meaning of a run difference | Requires inspection beyond the scalar OFV |
A common mistake is to say, “The replicated OFV is lower, so the replicated model is better.” That reasoning confuses a numerical direction with a valid comparison. The correction is to check first whether the runs use the same data, objective function, model, and estimation controls; only then can the difference be interpreted as a replication discrepancy. If those conditions do not hold, identify the changed basis and avoid treating the raw difference as evidence of improved fit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A submitted run has OFV 710.20 and a replication has OFV 710.08. Calculate \( \Delta_{\mathrm{OFV}} \) and state what its sign means.
- Two runs use different observation records but identical model code. Why is their OFV difference not a clean replication check?
- A lower replicated OFV occurs after a residual-error code change. What can and cannot be concluded from that difference?
- State two conditions needed before interpreting a difference as a likelihood-ratio statistic for nested models.
- What is a sensible next step when repeated runs with apparently identical inputs give noticeably different OFVs?
Solutions. (1) \(710.08-710.20=-0.12\); the replicated OFV is 0.12 lower. Whether this is acceptable depends on a justified tolerance and verification of matching run conditions. (2) Different records produce different likelihood contributions, so the runs do not evaluate the same data basis; inspect the records and rerun on a common dataset. (3) The replicated value is lower, but the runs use different model implementations. The difference alone does not show that the change is correct or that the model is better; compare the code with its specification and reproduce the submitted implementation. (4) The models must be nested and fitted to the same data using comparable likelihoods; the usual regularity conditions must also hold for the chi-square approximation. (5) Verify software, data, code, settings, starting values, and stochastic controls, then use controlled repeated runs to identify whether the discrepancy reflects run variability or an undocumented change.