Scaling Parameters for Stable SAEM Simulation

Why Scaling Matters in SAEM

In SAEM, the latent random effects are simulated conditionally on the observations, and population parameters are updated using the resulting information. Computation can become awkward when random effects have very different magnitudes or strong correlations, or when population parameters are measured on very different numerical scales. A proposal that moves effectively along one coordinate may barely move along another; an optimizer may likewise struggle when its numerical steps are unsuitable for the units of the parameters.

Scaling is a change of coordinates, not a license to change the model. If the transformation is invertible and the probability distribution and likelihood are transformed consistently, each point in the original model has a corresponding point in the scaled model. The predictions associated with corresponding points are identical. This tutorial focuses on fixed transformations that improve numerical behavior while retaining that equivalence.

Write a general NLME model as \(y_i\mid\eta_i,\theta\sim p(y_i\mid\eta_i,\theta)\), with \(\eta_i\sim N(0,\Omega)\). Here \(y_i\) denotes subject \(i\)'s observations, \(\theta\) the population parameters, and \(\Omega\) the random-effect covariance matrix. For the transformation to preserve the model, the conditional data distribution must use \(\eta_i=S z_i\), and the distribution of \(z_i\) must be the transformed distribution implied by \(\eta_i\). Merely rescaling the values passed to the simulator while leaving the target distribution unchanged is not a valid reparameterization.

Transforming Random Effects Correctly

For fixed nonsingular \(S\), the transformed random effect has mean zero and covariance \(S^{-1}\Omega S^{-\mathsf T}\), where \(S^{-\mathsf T}=(S^{-1})^\mathsf T\). In density form, the change of variables also contributes the determinant of \(S\). These facts ensure that the transformed density represents the same probability distribution as the original one.

If \(S\) is a Cholesky factor of a known covariance matrix, it can whiten the prior random effects: for \(\Omega=LL^\mathsf T\), choose \(S=L\), giving \(z_i\sim N(0,I)\). This can make a proposal scale that is sensible in standardized coordinates correspond to a correlated, scale-adapted move in the original coordinates.

There is an important qualification when \(\Omega\) is being estimated. A fixed reference matrix \(S\), perhaps chosen from initial values or a preliminary estimate, keeps the latent coordinates fixed during SAEM. Then the transformed covariance \(S^{-1}\Omega S^{-\mathsf T}\) must be updated as \(\Omega\) changes. By contrast, choosing \(S=L(\Omega)\) anew at each iteration makes the coordinates depend on the current parameter. That approach can be used only with a coherent treatment of the changing latent variables and transformed likelihood; it is not valid to keep treating all draws as if they were in one fixed coordinate system. A fixed scale is often the simpler and safer computational choice.

Scaling Conditional Proposals

Conditional simulation targets the random effects given the observations and current population parameters. A proposal can be constructed in transformed coordinates, but its density and the target density must refer to those same coordinates. For a fixed linear transformation, the Jacobian determinant is constant in \(z_i\) and cancels in a Metropolis-Hastings ratio when comparing two points under the same transformation. The transformed prior and the conditional data likelihood still both contribute to the target.

A symmetric random-walk proposal in \(z\), such as \(z'=z+\epsilon\) with \(\epsilon\sim N(0,h^2I)\), has a simple proposal ratio. In original coordinates, the corresponding increment is \(S\epsilon\), with covariance \(h^2SS^\mathsf T\). Thus, when \(S\) reflects the scales and correlations of the prior random effects, an isotropic step in \(z\) becomes a scale- and correlation-adapted step in \(\eta\). This does not guarantee good conditional mixing: the likelihood may produce posterior curvature quite different from the prior covariance. The earlier material on Metropolis-Hastings proposals and random-effect chain mixing covers acceptance and mixing assessment; scaling here changes the coordinates in which proposals are designed, not the conditional target.

Scaling Population Parameters for Numerical M-Steps

Population parameters can also be rescaled to make numerical optimization better conditioned. For an unconstrained parameter vector, an affine transformation \(\theta=c+D\xi\), with fixed nonsingular \(D\), gives a one-to-one correspondence between \(\theta\) and the scaled coordinate \(\xi\). The numerical M-step can optimize over \(\xi\) and map its result back to \(\theta\). With a differentiable objective \(Q(\theta)\), the gradient and Hessian in the new coordinates are \(D^\mathsf T\nabla_\theta Q\) and \(D^\mathsf T H_\theta D\), respectively. Suitable scales can reduce disparities in curvature that otherwise make numerical steps inefficient.

A positive parameter can instead be represented on a log scale, for example \(\log(\theta_j)=c_j+s_j\xi_j\), where \(s_j>0\). This ensures \(\theta_j>0\) for every real \(\xi_j\), but it is a nonlinear transformation, so gradients and optimization behavior must be understood on that transformed scale. In either case, parameter values in the model must be obtained by the stated inverse mapping. The model’s predictions at the mapped parameter values are unchanged; optimization may be numerically different, but the statistical model has not been altered.

Choosing a Transformation and Checking It

A useful transformation should be chosen from the scales, units, constraints, and dependence structure of the parameters being simulated or optimized. A diagonal \(S\) can standardize marginal random-effect scales; a full fixed matrix can additionally address correlation. Centering population parameters near plausible working values can make scaled coordinates easier to interpret. Transformations should be checked numerically and statistically, not chosen solely because the transformed variables look convenient.

A practical comparison is between unscaled coordinates, fixed standardization, and a parameterization that directly enforces constraints, such as a log scale for positive parameters. Unscaled coordinates are simplest but may be inefficient when numerical scales differ greatly. Fixed standardization can improve proposal geometry while retaining a fixed latent coordinate system, but requires consistent covariance and density calculations. A log transformation respects positivity, but changes the geometry of the optimization problem and may not solve correlation or poor identifiability. None of these options repairs a misspecified model or creates information absent from the data.

Diagnostics, Pitfalls, and Limitations

Scaling is worth revisiting if proposals still move very little in some directions, numerical M-steps take erratic or extremely small steps, or transformed covariance calculations lose symmetry or positive definiteness beyond expected rounding error. Also compare the model’s predictions at selected original and mapped parameter values: disagreement is evidence of an inconsistent transformation or implementation, not a harmless side effect of scaling. In SAEM, keep the coordinate definition and the interpretation of any accumulated statistics consistent; a statistic accumulated in one coordinate system cannot automatically be interpreted as though it had been accumulated in another.

• Changing the proposal but not its target. Scaling a proposal does not change the conditional distribution being sampled. The Metropolis-Hastings ratio must still use the correctly transformed target and proposal densities.
• Treating rounded whitening as exact. Rounded factors may not satisfy \(LL^\mathsf T=\Omega\). Check the factorization at working precision, or state and assess the approximation.
• Ignoring constraints. An affine transformation does not automatically enforce positivity. Use a suitable constrained or log parameterization and verify the mapped values remain valid.
• Assuming scaling guarantees mixing or convergence. Whitening the prior does not remove likelihood-induced posterior correlations, weak identification, or multimodality.

The central limitation is that scaling improves numerical representation, not the underlying statistical information. A poor reference scale can be ineffective, and an overly complicated transformation can introduce implementation errors. Preserve the exact model mapping, verify the transformed distribution, and judge a transformation by both its computational behavior and its equivalence to the original model.