Interpreting Objective Differences Across Approximations

Why an Objective Difference Needs Context

FO, FOCE, and Laplace can produce different objective values for the same data and model. A numerical difference is not self-interpreting: it may reflect a different approximation to the marginal likelihood, a different set of fitted parameters, or a different convention for constants in the likelihood. These explanations are distinct, and they support different conclusions.

The key first question is whether the values being compared were evaluated at the same parameter values. If they were, parameter re-estimation cannot explain their difference. If each method was separately optimized, the difference combines approximation and optimization effects. Before either comparison, verify that the methods use the same observations, model specification, random-effects parameterization, and objective convention.

Definition: Let $y$ denote the observed data and let $\theta$ collect the fixed-effect and random-effect distribution parameters. Write $\ell_A(\theta)$ for the approximate marginal log likelihood under method $A$, where $A$ is FO, FOCE, or Laplace. Under the common objective convention, $$ J_A(\theta)=-2\ell_A(\theta). $$ A smaller $J_A$ corresponds to a larger approximate log likelihood under that convention. The method-specific fitted estimate is denoted $\hat{\theta}_A$, and its minimized objective is $J_A(\hat{\theta}_A)$.

FO, FOCE, and Laplace use different approximations in evaluating the marginal likelihood: broadly, FO linearizes around the population random-effects value, FOCE uses conditional estimates, and Laplace uses local curvature around conditional modes. These methods can therefore give different objective functions even when they use the same model and data. A Laplace expression, for example, uses the log integrand at a conditional mode and a curvature term from its Hessian. This makes the approximation itself one possible source of an objective difference; it does not make the Laplace value automatically closer to the exact marginal likelihood.

This discussion assumes that each method is being used to evaluate a likelihood-based objective for the same observed-data target. Software implementations can differ in details such as included constants, parameter constraints, and the reported scaling of an objective. Check those details rather than treating labels such as “FOCE objective” as a guarantee that every convention matches.

Comparing Methods at Common Parameter Values

A fixed-parameter comparison evaluates all methods at one specified $\theta_0$. Since the parameter values are held constant, the difference isolates how the methods evaluate the objective at that point, subject to matching data and likelihood conventions. For methods $A$ and $B$,

J_B(\theta_0)-J_A(\theta_0) =-2\left\{\ell_B(\theta_0)-\ell_A(\theta_0)\right\}.

Thus, if $J_B(\theta_0)$ is lower than $J_A(\theta_0)$, method $B$ assigns a higher approximate log likelihood at $\theta_0$. This is a statement about the two approximations at those parameter values. It is not a test of which method is better, a measure of the approximation error relative to the exact likelihood, or evidence that the underlying model is true.

Worked Example: Reading a Fixed-Parameter Contrast

Suppose a hypothetical data set is analyzed with the same model, the same likelihood convention, and the same parameter vector $\theta_0$. At that vector, FO gives $J_{\mathrm{FO}}(\theta_0)=1240.0$, FOCE gives $J_{\mathrm{FOCE}}(\theta_0)=1216.0$, and Laplace gives $J_{\mathrm{Lap}}(\theta_0)=1210.0$. Find and interpret the contrasts.

For FOCE versus FO, $$ J_{\mathrm{FOCE}}(\theta_0)-J_{\mathrm{FO}}(\theta_0) =1216.0-1240.0=-24.0. $$ Because the objective is minus twice the approximate log likelihood, the corresponding log-likelihood contrast is $$ \ell_{\mathrm{FOCE}}(\theta_0)-\ell_{\mathrm{FO}}(\theta_0) =-\frac{-24.0}{2}=12.0. $$ At this parameter vector, FOCE gives an approximate log likelihood 12.0 units higher than FO.

For Laplace versus FOCE, $$ J_{\mathrm{Lap}}(\theta_0)-J_{\mathrm{FOCE}}(\theta_0) =1210.0-1216.0=-6.0, \qquad \ell_{\mathrm{Lap}}(\theta_0)-\ell_{\mathrm{FOCE}}(\theta_0)=3.0. $$ Laplace gives an approximate log likelihood 3.0 units higher than FOCE at $\theta_0$. The arithmetic supports these comparisons, but not a ranking of approximation accuracy: no exact marginal likelihood has been supplied for comparison.

Separating Approximation and Re-Estimation

When each method is optimized separately, the reported difference compares different objective functions at potentially different parameter estimates. An exact algebraic decomposition makes that distinction explicit. Let $\hat{\theta}_A$ be the estimate under method $A$ and $\hat{\theta}_B$ the estimate under method $B$. Add and subtract $J_B(\hat{\theta}_A)$:

\begin{aligned} J_B(\hat{\theta}_B)-J_A(\hat{\theta}_A) ={}&\left\{J_B(\hat{\theta}_A)-J_A(\hat{\theta}_A)\right\}\\ &+\left\{J_B(\hat{\theta}_B)-J_B(\hat{\theta}_A)\right\}. \end{aligned}

The first term compares the methods at the same parameter vector, $\hat{\theta}_A$; it is the approximation contrast at that reference point. The second term measures the change in method $B$'s objective when moving from $\hat{\theta}_A$ to $\hat{\theta}_B$. This identity is exact, not an approximation. If $\hat{\theta}_B$ is a global minimizer of $J_B$ over the same parameter space, the second term is at most zero. With only a local optimizer or an incomplete convergence, that guarantee may not hold.

Worked Example: Decomposing a Difference Between Minimized Objectives

Consider hypothetical results for method A = FO and method B = Laplace, all using the same data and objective convention. Suppose FO is minimized at $\hat{\theta}_{\mathrm{FO}}$, with $J_{\mathrm{FO}}(\hat{\theta}_{\mathrm{FO}})=98.0$. At that same parameter vector, Laplace gives $J_{\mathrm{Lap}}(\hat{\theta}_{\mathrm{FO}})=95.5$. Separately, Laplace is minimized at $\hat{\theta}_{\mathrm{Lap}}$, with $J_{\mathrm{Lap}}(\hat{\theta}_{\mathrm{Lap}})=93.8$.

The difference between the minimized objectives is $$ 93.8-98.0=-4.2. $$ Decompose it using the FO estimate as the reference point: $$ \begin{aligned} J_{\mathrm{Lap}}(\hat{\theta}_{\mathrm{Lap}}) -J_{\mathrm{FO}}(\hat{\theta}_{\mathrm{FO}}) ={}&\left(95.5-98.0\right) +\left(93.8-95.5\right)\\ ={}&-2.5-1.7=-4.2. \end{aligned} $$ The fixed-parameter Laplace-versus-FO contrast at the FO estimate is $-2.5$ objective units. A further $-1.7$ units comes from moving to the Laplace estimate while evaluating the Laplace objective. The total decrease of $4.2$ is therefore not solely an approximation contrast at common parameters.

This decomposition does not say that Laplace's fitted parameters are closer to a true parameter value. Nor does the lower Laplace minimum alone show that it provides a better approximation. Those conclusions require evidence beyond the two minimized objective values.

Likelihood Conventions and Comparability

An objective can include or omit additive terms that do not depend on the parameters. For example, a Gaussian log likelihood contains a normalizing constant. Omitting a common parameter-independent constant shifts the reported objective but does not change its optimizer. However, a comparison across methods is distorted if one value includes the constant and another omits it.

More generally, if a reported objective is $J_A^*(\theta)=J_A(\theta)+c$, where $c$ is the same parameter-independent constant for every method being compared, then every reported value shifts by $c$. Differences calculated under that same convention are unchanged. If the conventions differ, raw objective differences include the convention mismatch. Terms that depend on parameters cannot be dismissed this way: omitting them can change both the objective surface and its minimizer.

Worked Example: Adjusting for an Omitted Gaussian Constant

Suppose 20 hypothetical observations have Gaussian residual contributions. One reported objective includes the term $n\log(2\pi)$, while another omits it. For this comparison, all other terms, data, and parameter values are assumed to match. The constant shift in the objective is $$ n\log(2\pi)=20\log(2\pi) =20(1.837877)=36.7575. $$

Suppose FOCE reports $500.0$ with the constant included and Laplace reports $463.2$ with it omitted. Converting Laplace to the full-constant convention gives $$ J_{\mathrm{Lap,full}}=463.2+36.7575=499.9575\approx 500.0. $$ The apparent raw difference, $463.2-500.0=-36.8$, is almost entirely explained by the omitted constant. After aligning conventions, the difference is about $-0.0425$ objective units, using the displayed rounded inputs. It would be misleading to interpret the raw $-36.8$ as an approximation effect.

This adjustment is valid only under the stated assumption that the omitted term is parameter-independent and that the same observations and likelihood contributions are being compared. A discrepancy in the observations, residual model, or parameter-dependent terms requires more than adding a constant.

A Practical Interpretation Sequence

Use the following checks before interpreting any objective difference. They are especially useful when values come from separate runs or software outputs.

Align the target.
Confirm that the same observations, model, residual-error specification, random-effects structure, and parameterization are used. Otherwise, the comparison is not solely about FO, FOCE, or Laplace.

Align the convention.
Check objective scaling, included likelihood constants, and any other reporting conventions. Convert values to a common convention before computing contrasts.

Identify the evaluation point.
If the parameter vector is common, interpret the difference as an approximation contrast at that point. If each value is minimized separately, distinguish the fixed-point contrast from the parameter-movement contribution.

Check the fits.
Review convergence status, boundary estimates, and whether the reported solution is a credible minimum. Failed optimization can make a nominally minimized objective unreliable.

Limit the conclusion.
Report what the objective values establish: relative approximate likelihood under stated conditions. Do not infer approximation accuracy, parameter truth, or statistical significance from the difference alone.

What Objective Differences Can and Cannot Establish

The main alternative to a fixed-parameter comparison is to compare separately optimized objectives. The first isolates the methods' evaluations at a common point; the second describes the best reported fit under each method, but mixes approximation and re-estimation. Neither comparison by itself identifies which approximation is closest to the exact marginal likelihood. A reference calculation or other validation is needed to make an accuracy claim.

Objective differences also are not automatically test statistics. In particular, a difference between FO, FOCE, and Laplace values does not become a likelihood-ratio test merely because it is expressed using $-2\ell$. A likelihood-ratio interpretation requires a suitable comparison of models and the conditions for its reference distribution; methods that use different approximations do not, by that fact alone, meet those requirements. Likewise, an objective difference does not give a p-value or a probability that one method is correct.

Diagnostics matter because an apparent method contrast can instead signal a failed or incomparable calculation. Warning signs include different observation counts, parameter estimates at unexpected boundaries, optimizer failure, inconsistent objective scaling, or changes in the model specification between runs. Even when these checks pass, differences can be small relative to numerical precision, so avoid overinterpreting digits beyond the precision reported by the fitting procedure.

Common Mistake

Wrong reasoning: “Laplace has a lower minimized objective than FO, so Laplace is more accurate.”
Correction: A lower objective means a higher reported approximate log likelihood under the stated convention. The difference may include both a change in approximation and a change in fitted parameters. It does not measure error relative to the exact marginal likelihood. First align the model and likelihood conventions; then, if the fits are separately optimized, separate the fixed-parameter and parameter-movement contributions.

Key takeaway: Always state whether an objective contrast is evaluated at common parameter values or after separate optimization. A fixed-point contrast compares approximation values at that point; a minimized-objective contrast also reflects parameter movement. Both require matching likelihood conventions, and neither alone ranks approximation accuracy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

At the same parameter vector and under the same convention, FO gives an objective of 220 and FOCE gives 214. What is the approximate log-likelihood difference, FOCE minus FO?
In one sentence, explain why a lower separately minimized Laplace objective than FO objective does not isolate an approximation effect.
A method omits a parameter-independent constant of 12 objective units. How does adding that constant affect its parameter estimate, assuming the optimizer is unchanged?
Name two checks to make before comparing objective values from separate runs.

Solutions. 1. The objective difference is $214-220=-6$, so the approximate log-likelihood difference is $-(-6)/2=3$: FOCE is higher by 3 log-likelihood units at that vector. 2. The methods may be evaluated at different parameter estimates, so the difference combines the approximation contrast and parameter movement. 3. Adding a parameter-independent constant shifts every objective value by 12 but does not change the minimizer. 4. For example, confirm the same data and model, and verify the same objective scaling and likelihood constants; also check that both fits converged successfully.