Diagnosing Failure of the First-Order Approximation

Why Diagnose the Approximation?

First-order information approximations replace each subject’s nonlinear prediction by a local linear function of that subject’s random effects. This can make population-design calculations tractable, but the resulting information is only as representative as that local approximation. A design may look informative under the linearized model while the nonlinear predictions vary materially over plausible random effects.

The central diagnostic question is not simply whether a prediction function is nonlinear. It is whether its curvature, over the random-effect values relevant to the design, is large enough to matter relative to residual noise and the random-effect-driven variation. The checks here use local derivatives and prediction residuals to answer that question. They flag potential failure; they do not, by themselves, prove that a particular Fisher matrix is inaccurate.

For the local expansion to be useful, predictions must be sufficiently smooth near zero, the nominal parameters and design must be fixed for the diagnostic, and the random-effects distribution must place most of its probability in a region where the expansion is informative. Residuals are commonly assumed to have conditional mean zero; their variance or covariance supplies the measurement-error scale against which prediction discrepancies can be assessed. These are diagnostic assumptions, not guarantees that the model is correctly specified.

Curvature and a Scale-Aware Check

For a scalar prediction and a vector of random effects, a second-order Taylor expansion gives

where \(H_{ij}\) is the Hessian matrix of second derivatives at zero and \(r_{ij}\) contains terms of third and higher order. The quadratic term is the leading local discrepancy from first-order linearization. If there is one random effect, it is \(\tfrac12 H_{ij}\eta_i^2\), not \(H_{ij}\eta_i^2\). This factor of one-half matters in every numerical check.

For centered random effects with covariance \(\Omega\), the expected quadratic discrepancy is

To see this, write the quadratic form as a sum of terms: \(E(\eta_i^{T}H_{ij}\eta_i)=\sum_{a,b}H_{ij,ab}E(\eta_{i,a}\eta_{i,b})\). Since \(E(\eta_i\eta_i^{T})=\Omega\), this sum is \(\operatorname{tr}(H_{ij}\Omega)\). Multiplying by one-half gives the stated result. Thus, even when the random effects have mean zero, curvature can shift the mean prediction away from the prediction at zero random effects.

A useful initial scale comparison, when the residual variance at observation \(j\) is \(R_{jj}\), is

This ratio measures the magnitude of the expected quadratic shift in residual standard-deviation units. It is not a universal pass/fail threshold: a small mean shift can coexist with substantial subject-level curvature, and a large value signals a discrepancy worth investigating rather than automatically invalidating the design.

The expected shift is only one summary. If the distribution of quadratic corrections matters, a Gaussian random-effects assumption gives a second-moment check. For symmetric \(H_{ij}\),

This expression follows by expanding the squared quadratic form and using the centered Gaussian fourth-moment identity \(E(\eta_a\eta_b\eta_c\eta_d)=\Omega_{ab}\Omega_{cd}+\Omega_{ac}\Omega_{bd}+\Omega_{ad}\Omega_{bc}\). It captures both a nonzero average curvature shift and variation around that shift. In practice, simulation from the specified random-effects distribution can provide an easier check, especially when the Taylor remainder is not negligible.

Worked Checks Across Random-Effect Values and Times

Random-effect magnitude matters because the local approximation is evaluated over a distribution, not only at \(\eta=0\). A small Hessian may still produce material discrepancies when the random-effect variance is large; conversely, noticeable curvature may be harmless if plausible random effects are tightly concentrated near zero and residual noise is large. Inspect curvature in standardized coordinates as well as in the original parameterization. For example, writing \(\eta=\Omega^{1/2}z\), with \(z\) having identity covariance under a Gaussian model, makes the curvature matrix in standardized coordinates \(\Omega^{1/2}H\Omega^{1/2}\). This combines curvature and random-effect scale rather than judging either alone.

Residual Diagnostics and Failure Modes

A practical check is to simulate random effects and observations from the proposed model at the planned design points, calculate both nonlinear and first-order predictions, and compare their differences with the residual scale. When simulated random effects are known, the observation-minus-first-order-prediction residual can be separated into the linearization discrepancy and observation error. In data analysis, random effects are not known; residual plots then combine approximation error with estimation and model error, so they are evidence to investigate rather than a direct measurement of the Taylor remainder.

Useful plots and summaries include residuals against time, predicted value, covariates, and the random-effect values used in simulation. Curvature can create systematic patterns: a quadratic effect in a centered random effect contributes in the same direction at positive and negative values, while higher-order terms can create asymmetry. Patterns that grow at particular sampling times may indicate that linearization is less adequate there. The design matters: a prediction that is nearly linear at one visit can be substantially curved at another.

• Large random effects: if plausible values extend far from zero, a local derivative at zero may not represent predictions across the population. Inspect the full nonlinear-minus-linearized differences over simulated or otherwise plausible values.
• Curvature hidden by a mean check: positive and negative discrepancies can average out across observations or visits. Examine individual prediction differences and their spread, not only their mean.
• Residual patterns attributed too quickly: structured residuals may reflect an inadequate structural or residual model rather than failure of linearization. Compare nonlinear predictions and assess alternative explanations.
• Unstable numerical derivatives: Hessians computed with unsuitable finite-difference steps can be noisy. Check derivative stability over step sizes, compare with automatic differentiation when available, and verify selected predictions directly.
• Ignoring the residual scale: curvature in concentration units has no universal meaning. Compare it with the residual standard deviation at the same observation, and use consistent units.

Choosing a Follow-Up Check

First-order, FOCE, and exact marginal approaches answer related but not identical questions. A first-order calculation uses derivatives at zero random effects. FOCE evaluates a local approximation around conditional estimates of individual random effects, which can better represent some subjects when those conditional estimates are informative. Exact marginal information accounts for integration over the random-effects distribution rather than relying on the same local expansion. The comparison of these approximations is useful context; this tutorial’s diagnostics are intended to identify when the first-order expansion warrants closer checking.

When local curvature is small over plausible random effects and prediction differences are small relative to residual variation, first-order results may be adequate for the intended design comparison. When the discrepancy is material, a sensible follow-up is to compare predictions and information under the nonlinear model using numerical integration or simulation, and to assess whether FOCE changes the conclusion. A more computationally demanding method is not automatically correct if the structural model, covariance assumptions, or numerical calculations are poor.

These diagnostics have limits. Taylor derivatives are local, while the random-effects distribution may extend into regions where higher-order terms dominate. A residual-scale ratio does not quantify all changes to the population Fisher matrix, and a small discrepancy at a nominal parameter value does not establish adequacy under different parameter values or designs. Diagnostics should therefore be repeated at relevant sampling times and across plausible model conditions. They are strongest when derivative-based checks, direct nonlinear comparisons, and residual-pattern checks tell a consistent story.