FOCE Interaction and Eta-Dependent Residual Variance

Why Residual-Variance Interaction Matters

In a nonlinear mixed-effects model, an individual random effect can change the predicted response. If the residual error variance is itself a function of that prediction, the random effect also changes the scale of the observation error. The random effect then interacts with the residual error: individuals with different eta values can have different residual variances, even at the same nominal observation time.

This matters when forming a subject-level likelihood approximation. A conditional linearization that updates the predicted mean but holds the residual variance fixed omits part of the way eta affects the data. FOCE interaction retains the dependence of residual variance on eta in the conditional likelihood. It does not mean adding another random effect, nor does it select a residual error model; it refers to how the chosen error model is carried into the approximation.

The usual conditional observation model is \(y_i\mid\eta_i\sim N(f_i(\eta_i),R_i(\eta_i))\). Its conditional log likelihood, excluding the random-effect density, is

where \(m_i\) is the number of observations for subject \(i\). The conditional log posterior used to locate the individual mode adds the random-effect log density:

with \(q\) the dimension of \(\eta_i\). The conditional mode \(\hat\eta_i\) maximizes this joint quantity. The formulas assume a correctly specified conditional normal observation model, a positive-definite \(\Omega\), and a positive-definite \(R_i(\eta_i)\) in the region being evaluated. These conditions ensure the stated densities and inverses exist.

What Changes in the Local Approximation

Write \(\delta_i=\eta_i-\hat\eta_i\), and let \(J_i\) be the Jacobian of \(f_i\) with respect to \(\eta_i\), evaluated at \(\hat\eta_i\). Conditional linearization gives \(f_i(\eta_i)\approx f_i(\hat\eta_i)+J_i\delta_i\). Under a noninteraction treatment, the covariance is held at \(R_i(\hat\eta_i)\). Under FOCE-I, the conditional likelihood instead retains \(R_i(\eta_i)\), or locally represents its variation by a first-order expansion around \(\hat\eta_i\):

Thus the corresponding local subject-level integrand has a linearized mean and an eta-varying covariance:

Here \(\phi\) and \(\phi_{m_i}\) denote the normal densities of the indicated dimensions. The approximation to the mean is local; retaining the original positive covariance function is often safer than extending a linear covariance expansion far from the mode. With covariance frozen at \(R_i(\hat\eta_i)\), the linearized mean and Gaussian random-effect distribution yield a Gaussian convolution. When covariance varies with eta, the integrand is a mixture of normal densities with changing scales and generally does not reduce to that same single Gaussian convolution.

The distinction can also be seen from the conditional score. For one random-effect coordinate \(\eta\), write \(r=y-f(\eta)\), \(R=R(\eta)\), \(f'=\partial f/\partial\eta\), and \(R'=\partial R/\partial\eta\). Differentiating the conditional data log likelihood gives

The first term is the mean contribution. The second and third terms arise because both the quadratic residual penalty and the log determinant change as the covariance changes. The prior adds \(-\eta^T\Omega^{-1}\) in the multivariate score. A noninteraction score omits the covariance-derivative terms; therefore, unless those terms are zero or negligible, it can locate a different conditional mode.

Worked Calculations

Assumptions, Diagnostics, and Alternatives

The interaction correction is relevant when the selected residual variance model actually depends on eta. For example, under proportional error with standard deviation \(\sigma_{\mathrm{prop}}f_j(\eta)\), observation \(j\) has variance \(\sigma_{\mathrm{prop}}^2f_j(\eta)^2\). With combined additive and proportional error, \(v_j(\eta)=\sigma_{\mathrm{add}}^2+\sigma_{\mathrm{prop}}^2f_j(\eta)^2\). In the latter case, \(v_j'(\eta)=2\sigma_{\mathrm{prop}}^2 f_j(\eta)f_j'(\eta)\). By contrast, a purely additive error model with constant variance has \(R_i'(\eta)=0\), so this particular interaction term is absent.

A practical check is to evaluate whether the variance changes meaningfully over the eta range that contributes to the subject likelihood, not only at \(\eta=0\). Inspect the fitted variance function and conditional predictions, verify that covariance matrices remain positive definite, and check numerical stability of the individual modes and objective as tolerances or starting values change. Large differences between interaction and noninteraction fits can indicate that the variance dependence matters, but can also signal weak information, an unsuitable residual model, or poor local approximation. The difference alone does not establish which fit is adequate.

A first-order approximation in eta may be poor if the prediction or variance is strongly curved near the mode, if the mode is far from the population mean, or if the data are sparse enough that the individual likelihood is broad. A linearized covariance can also become nonpositive definite away from the expansion point, even when the original variance model is valid. In such cases, retain and evaluate a valid covariance function where possible, assess sensitivity to the approximation, and consider other marginal-likelihood approximations. Those approaches have their own assumptions and numerical costs; FOCE-I is not a guarantee of an exact marginal likelihood.

The comparison is about the approximation, not which residual model is intrinsically preferable. The introductory material on marginal likelihood as an integral explains why the subject contribution integrates over eta; FOCE-I changes the local integrand used to represent that contribution. It does not remove the need to specify and assess the observation model.

Common Pitfall

A common mistake is to say, “Proportional error is handled because the prediction was linearized, so the residual variance is automatically handled too.” That is incorrect: linearizing \(f_i(\eta_i)\) updates the conditional mean, but does not itself retain \(R_i(\eta_i)\). The correction is to write the conditional likelihood with its log-determinant and residual quadratic term, then check whether both depend on eta. If they do, their derivatives contribute to the conditional score and their variation belongs in the interaction approximation.