Fixed-Effect Bias Across FO, FOCE, and Laplace

Why Compare Fixed-Effect Bias?

FO, FOCE, and Laplace approximate the integration over subject-specific random effects in different ways. When the individual model is nearly linear in those effects, their fixed-effect estimates may be similar. As nonlinearity increases, an approximation can represent the marginal distribution less accurately, shifting the estimated population parameters. A controlled simulation makes that shift measurable while distinguishing it from ordinary sampling variation.

The target here is fixed-effect bias, not which method has the smallest objective value or the best estimate of every model parameter. The generating fixed effect is known in a simulation. Across repeated data sets, the average estimate can therefore be compared directly with that known value. A single estimate from one data set cannot by itself establish bias.

Bias is a property of a method under a specified data-generating process, sample size, and estimation setup. It is not the difference between one fitted estimate and the truth: that difference also includes the random sampling error in that particular data set. The simulation must therefore repeat data generation and fitting, not merely fit three methods once.

A Controlled Nonlinear Model

Consider one response per subject and a scalar random effect:

The random effects and residual errors are independent across subjects and of each other. The exponential response model is correctly specified; the parameter \(\kappa\) controls how strongly the response changes with the random effect. At \(\kappa=0\), the random effect does not affect the response. Increasing \(\kappa\) increases the curvature of the individual response as a function of \(\eta_i\). The scalar \(\theta_0\) is the generating fixed effect, and \(\omega^2\) and \(\sigma^2\) are the random-effect and residual variances.

For a focused comparison, hold \(\omega^2\) and \(\sigma^2\) fixed at their generating values during estimation, and estimate \(\theta\). This isolates how the approximations affect the fixed-effect estimate; it does not represent every practical fitting problem, where variance parameters are usually estimated as well. Use the same sample size, parameter values, and data sets for all three methods at each value of \(\kappa\). Only \(\kappa\) changes across conditions.

What the Three Approximations Do

Write an individual marginal likelihood contribution as an integral over the random effect. Let \(h_i(\eta)\) denote the log of the conditional response likelihood multiplied by the random-effect density. For example, \(h_i(\eta)=\log p(y_i\mid\eta;\theta)+\log p(\eta)\). The integral is \(\int \exp\{h_i(\eta)\}\,d\eta\). The methods differ in how they approximate this contribution.

• FO linearizes the individual response model around \(\eta=0\). Its local approximation uses the response and its derivative at zero, so it does not adapt that expansion point to the observed subject response.
• FOCE linearizes around an individual conditional mode, the value of \(\eta\) that maximizes the individual log posterior \(h_i(\eta)\). The expansion can therefore adapt to the subject's data, although it still uses a local linear approximation to the response.
• Laplace approximates the log integrand quadratically around its conditional mode. If \(\hat{\eta}_i\) is the mode and \(H_i=-\partial^2 h_i(\hat{\eta}_i)/\partial\eta^2\) is positive, its one-dimensional approximation is \(\exp\{h_i(\hat{\eta}_i)\}(2\pi)^{1/2}H_i^{-1/2}\).

FOCE and Laplace are not interchangeable labels: FOCE linearizes the response, whereas Laplace uses the local curvature of the log integrand. Nor does a local approximation guarantee zero bias. Its accuracy depends on how well the local representation captures the integrand over the region that contributes to the integral.

Why Nonlinearity Can Shift an Estimate

The mechanism is visible from a Taylor expansion around zero. For a smooth individual response \(f(\theta,\eta)\), a second-order expansion contains a curvature term:

For a centered random effect, the expectation of the linear term is zero, but the expectation of the quadratic term is generally not: \(E(\eta^2)=\omega^2\). A first-order expansion at zero drops that contribution. When curvature is small, the omitted term may have little effect; when curvature grows, it can materially change the marginal distribution used to estimate \(\theta\). This is a justification for testing sensitivity to nonlinearity, not a theorem that FO must be biased in a particular direction for every model.

Separate Bias From Simulation Noise

An empirical bias is itself estimated from a finite number of replicate data sets. If the standard deviation of the replicate estimates is \(s_{\hat{\theta}}\), the Monte Carlo standard error of their average is

This standard error measures the simulation noise in the estimated average; it is not the standard error of an estimate from one data set. Increasing the number of independent replicates reduces Monte Carlo error, but does not remove bias inherent in the method under the chosen setup.

Design Checks, Diagnostics, and Limits

A useful comparison changes one feature at a time. Keep the generating parameter values, subjects per data set, number of replicates, and fitting constraints fixed across methods; use the same replicate data sets for FO, FOCE, and Laplace so differences are not needlessly mixed with different simulated samples. Check that each method converges, that the fixed-effect estimates are on the intended scale, and that unusual failures or extreme estimates are not silently discarded. Report the number and handling of failed fits, since selective removal can alter the estimated bias.

The main patterns to inspect are empirical bias by \(\kappa\), variability across replicates, and the Monte Carlo standard error of each average. If an apparent difference between methods is small compared with Monte Carlo error, increase the number of replicates before interpreting it. If a method's average shifts while its replicate variability is large, report both: a mean alone can obscure unstable estimation. Also verify the implementation and parameterization, since a coding error or optimizer failure is not approximation bias.

The comparison has important limits. Fixing variance parameters isolates one source of difference but does not show the bias behavior when variance parameters are estimated jointly. A result from the exponential model does not establish a ranking for other response functions, random-effect distributions, sampling schedules, or residual-error models. FOCE and Laplace can reduce bias in some conditions, but neither is guaranteed to be unbiased or more accurate in every finite sample. Where feasible, a more direct numerical evaluation of the marginal likelihood can serve as a benchmark, but the benchmark itself should be checked for numerical accuracy.

Common Mistakes

• Calling one estimate's error “bias.” One estimate minus the generating value is a realized estimation error. Bias is an average over repeated data sets.
• Confusing the fixed effect with the marginal response mean. In the example, \(\theta_0=10\) response units, while the population mean response at \(\kappa=1\) is about 10.83 response units. They are different quantities.
• Concluding that Laplace always wins. A closer average in one controlled setup supports a conclusion about that setup only. Change the model or data conditions and the ordering may change.
• Ignoring simulation precision. A difference between average estimates is not persuasive if it is comparable to Monte Carlo noise or driven by failed fits.