Additive Versus Proportional Error in FO and FOCE

Why the Residual-Error Model Changes the Approximation

FO and FOCE approximate the contribution of a subject by simplifying the integration over that subject’s random effects. The residual-error model matters because it determines not only how far an observation is from its prediction, but also how much residual variation is assigned to that observation. With additive error, the residual variance is constant as the random effects change. With proportional error, the variance changes with the prediction, which commonly depends on the random effects. An approximation that ignores this dependence can therefore alter the subject likelihood, not merely its description of the mean.

This tutorial compares the two error structures at the level of the conditional likelihood and explains their implications for FO and FOCE. It focuses on the interaction between random effects and residual variance; it does not treat an approximation as a substitute for checking whether the chosen residual model is scientifically appropriate.

Under either model, the subject likelihood is the integral of the conditional observation density times the random-effect density. The integral is difficult when \(\mu_{ij}(\eta_i)\), or the residual variance, depends nonlinearly on \(\eta_i\). For the formulas below, write \(v_{ij}(\eta_i)\) for the conditional residual variance. The conditional log density, omitting no parameter-dependent terms, is

The log of the joint integrand adds the Gaussian random-effect log density to \(\ell_i(\eta_i)\). Under additive error, \(v_{ij}(\eta_i)=\sigma_a^2\), so its derivative with respect to any random effect is zero. Under proportional error, \(v_{ij}(\eta_i)=\sigma_p^2\mu_{ij}(\eta_i)^2\). Thus, for a scalar random effect \(\eta\),

Unless the prediction is insensitive to \(\eta\), this derivative is not zero. The likelihood changes through both the residual \(r_{ij}\) and the variance assigned to that residual.

What FO and FOCE Approximate

For a clear comparison, use the following convention. FO linearizes the prediction around the population random-effect value \(\eta_i=0\) and evaluates the residual variance there. If \(J_i\) is the matrix of prediction derivatives at zero and \(R_i(0)\) is the diagonal matrix of residual variances at zero, the resulting marginal approximation is normal with mean \(\mu_i(0)\) and covariance \(J_i\Omega J_i^\mathsf{T}+R_i(0)\). This is a local approximation: it does not reproduce the full variation of a random-effect-dependent residual variance over the distribution of \(\eta_i\).

FOCE instead locates a conditional mode for each subject and makes a local approximation around that subject-specific value. FOCE with interaction retains the dependence of the conditional residual likelihood on \(\eta_i\), including variance dependence when the error model makes the variance random-effect dependent. A variance-frozen, non-interaction approximation holds the residual variance fixed in the local likelihood approximation. It can still use a variance evaluated at a subject-specific value; what it omits is the local change in that variance as \(\eta_i\) changes.

These descriptions specify the feature being compared, rather than asserting that every software implementation uses identical computational details. In particular, “with interaction” concerns the random-effect dependence in the residual likelihood. It does not mean that the residual errors are correlated with one another.

Why the Interaction Appears: A Short Derivation

For one observation and a scalar \(\eta\), differentiate the conditional log density. Since \(r=y-\mu(\eta)\), \(dr/d\eta=-d\mu/d\eta\). Applying the product and chain rules to the log-density expression gives

where primes denote derivatives with respect to \(\eta\). The term involving \(2rr'/v\) describes the change in residual fit as the prediction moves. The terms involving \(v'\) describe the change in the log variance and in the residual’s variance-scaled contribution. They vanish when \(v\) is constant, but generally remain under proportional error. A local approximation that freezes \(v\) discards these variance-change terms; retaining them is the interaction distinction relevant here.

FO Versus FOCE in a Proportional-Error Model

The FO covariance formula makes the simplification especially visible. For \(\mu(\eta)=10\exp(\eta)\), a first-order expansion at zero is \(\mu(\eta)\approx10+10\eta\). Suppose \(\eta\sim N(0,0.25)\). With either error model set to variance 4 at \(\eta=0\), FO gives an approximate marginal distribution for the observation with mean 10 and variance

Thus this FO approximation is \(Y\approx N(10,29)\) under both models for these parameter values. This does not say the models have the same conditional likelihood away from zero. It says that, under the stated FO convention, their residual variance is evaluated at zero, where both were set to 4. In the proportional model, the true conditional variance changes with \(\eta\), a feature that this FO covariance does not carry through the integration.

FOCE targets a subject’s local region around its conditional mode rather than imposing a population-value expansion for every subject. If the mode is away from zero, the proportional variance evaluated there may differ substantially from its value at zero. FOCE with interaction also accounts locally for how that variance changes around the mode. This can change the local curvature and the approximate subject contribution. It is not guaranteed to move the estimate or objective in one particular direction: the result depends on the residuals, prediction derivatives, random-effect distribution, and the other observations for that subject.

Assumptions, Diagnostics, and Limitations

The comparisons above assume Gaussian conditional residuals, conditional independence across observations given the random effects, and a Gaussian random-effect distribution with covariance \(\Omega\). The proportional model as written requires positive predictions; if predictions can be zero or negative, \(\sigma_p^2\mu^2\) may not provide an adequate error specification, and the model needs reconsideration. These assumptions define the likelihood being approximated; FO or FOCE does not make them true.

Useful warning signs include residual spread that increases with prediction under a fitted additive-error model, substantial changes in predictions across plausible random effects, or a large difference between proportional variances at zero and near subject-specific modes. These indicate that evaluating variance only at zero may be a consequential simplification. Also inspect whether local approximations are stable as starting values or model parameterizations change, and whether a few subjects with large residuals or extreme conditional modes dominate the objective. Such patterns are prompts to investigate the model and approximation, not automatic proof that either is wrong.

Additive error is a natural alternative when residual variability is approximately constant on the observation scale. Proportional error is useful when residual standard deviation tends to scale with prediction. A combined additive-plus-proportional variance model is another possible error specification, but selecting it requires justification from the data and scientific setting. These are model choices, distinct from the choice between FO and FOCE. Laplace approximation is a further integration approximation centered on conditional modes and local curvature; it is an alternative computational treatment, not a replacement for choosing a defensible residual-error model.

A common mistake is to say, “The proportional variance is 4 at the population prediction, so interaction is irrelevant.” The correction is that this equality only compares the models at that prediction. If \(\mu_{ij}(\eta_i)\) changes with \(\eta_i\), then \(v_{ij}(\eta_i)\) can change too, and its derivative can contribute to the conditional likelihood. Conversely, proportional error does not guarantee an important interaction: if predictions barely change over the relevant random-effect region, the variance change may be negligible.