Reparameterization and Scaling for Stable Optimization

Why Parameter Coordinates Matter

Selection-model likelihoods can be difficult to optimize for reasons that are distinct from the statistical model itself. Predictors may have very different units, a variance parameter must remain positive, and an outcome effect in the response model may be numerically small simply because the outcome is measured on a large scale. These features can give an optimizer coordinates with very different numerical ranges or curvature.

A reparameterization changes the coordinates used to describe the same model. When the transformation is one-to-one over the allowed parameter space, every parameter value in one coordinate system corresponds to exactly one value in the other. The fitted likelihood and implied probabilities do not change; what can change is how easily a numerical algorithm searches the parameter space.

For a concrete selection-model setting, let \(R_i=1\) indicate that outcome \(Y_i\) is observed, let \(x_i\) denote predictors in the outcome model, and let \(z_i\) denote predictors in the response model. A common structure has an outcome distribution indexed by regression parameters and a variance, together with a response probability such as

Here \(\alpha\) is the outcome-dependent response effect, measured per unit of \(Y_i\). The transformations in this tutorial do not assert that this particular link or outcome distribution is required. They apply whenever the model has the corresponding linear predictors and parameter constraints. The observed-data likelihood, including any required integration over unobserved outcomes, remains the likelihood being optimized.

Centering and Scaling Predictors

Suppose a linear predictor contains an intercept and a coefficient for a predictor \(x\):

Choose fixed constants \(c\) and \(s>0\), and define \(x^*=(x-c)/s\). Substituting \(x=c+sx^*\) gives the identical linear predictor in the new coordinates:

Thus the transformed intercept is the linear predictor at \(x=c\), and the transformed slope is the change in the linear predictor for an increase of \(s\) original units in \(x\). Centering often makes the intercept refer to a typical predictor value rather than \(x=0\); scaling can bring a slope to a numerically convenient magnitude. The constants should be fixed from the covariate scale, not treated as additional parameters unless the model specifically requires that.

The same algebra applies to predictors in the outcome model and to multiple predictors, one at a time. If an outcome mean is \(\mu_i=\beta_0+\beta_1x_i\), then replacing \(x_i\) by \(x_i^*\) requires \(\beta_0^*=\beta_0+c\beta_1\) and \(\beta_1^*=s\beta_1\). If the predictor has zero variation, it cannot be divided by a positive empirical standard deviation; its slope is not separately identified from the data. Scaling does not repair that lack of information.

Positive Variance Parameters

A standard deviation \(\sigma\) must be positive. Directly optimizing over \(\sigma\) requires an algorithm that respects the constraint, and trial steps can otherwise leave the parameter space. A common alternative is the unconstrained coordinate \(\lambda=\log\sigma\), so that \(\sigma=\exp(\lambda)\) for every real \(\lambda\). The variance is then \(\sigma^2=\exp(2\lambda)\). This is the log standard deviation, not the log variance.

The chain rule also changes the numerical score. If the likelihood as a function of \(\sigma\) is \(\ell(\sigma)\), then, with \(\sigma=\exp(\lambda)\),

Consequently, gradients and curvature are expressed in different units after reparameterization. This can make steps more suitable for the parameter’s positive, multiplicative scale. It is not a claim that the transformed likelihood contains more information.

Scaling an Outcome Effect in the Response Model

An outcome-dependent response effect can be especially awkward numerically when the outcome is measured in large units. In the logistic model above, \(\alpha\) is the change in response log-odds per one outcome unit. If one unit is small relative to the spread of \(Y\), the numerical value of \(\alpha\) may be small. Centering and scaling \(Y\) changes that coefficient’s units, but the response intercept must change as well.

Let \(u=(y-c)/s\), with fixed \(s>0\), and write the original linear predictor as \(\gamma_0+\alpha y\). Since \(y=c+su\), the equivalent expression is

The transformed response effect \(\delta_1\) is per \(s\) outcome units. This is an invertible transformation: \(\alpha=\delta_1/s\) and \(\gamma_0=\delta_0-\delta_1c/s\). Keeping the old intercept while changing only the outcome scale would generally change the model’s response probabilities.

Why Curvature and Conditioning Change

A numerical optimizer works with derivatives in its chosen coordinates. For an affine transformation \(\theta=A\phi+b\), where \(A\) is an invertible matrix, the Hessian of the re-expressed log likelihood is

This follows by applying the chain rule to the gradient, \(\nabla_{\phi}\ell=A^\mathsf{T}\nabla_{\theta}\ell\), and differentiating once more, since \(A\) is constant. For a nonlinear transformation \(\theta=f(\phi)\), the Hessian also has terms involving the gradient in \(\theta\) and the second derivatives of \(f\); at a stationary point those gradient terms vanish. These relationships explain why the apparent curvature, and hence numerical conditioning, depends on coordinates.

Practical Checks, Alternatives, and Limitations

A useful implementation begins by writing down both directions of each transformation. After optimization, map estimates back to the original scientific units before interpreting coefficients or reporting results. Then check that the original-scale and transformed-scale calculations give the same linear predictors, response probabilities, and observed-data log likelihood at corresponding parameter values. For an iterative fit, also check that transformed parameters remain finite and that any variance recovered by exponentiation is positive and numerically representable.

Compare gradients and Hessian diagnostics in the optimization coordinates, but do not read improved conditioning as evidence that the model is identified. Scaling can help an algorithm take better-sized steps; it cannot create information that the data do not contain or eliminate a likelihood ridge. When the likelihood remains weakly curved in a direction after sensible scaling, that may reflect genuine weak identification rather than poor units. The earlier material on Hessian diagnostics and weak identification distinguishes these issues.

Scaling is one option among several. An optimizer may also allow parameter-specific step scales, bounds, or constrained optimization; these can address algorithm mechanics without changing the scientific parameterization. Profiling nuisance parameters can clarify a likelihood’s shape in a parameter of interest, but it does not replace careful numerical optimization. Reparameterization is particularly useful for positivity constraints and disparate units, while optimizer scaling can be convenient when estimates must remain directly in their original coordinates. Neither approach repairs an incorrect likelihood, inaccurate numerical integration, a flat ridge, or multiple relevant maxima.

A disciplined workflow is therefore: choose fixed centers and scales; derive the exact inverse mapping; optimize in the new coordinates; verify likelihood and predictions under the mapping; and interpret estimates only after returning them to their original units. A successful coordinate change improves computation while preserving the model. If likelihood values or implied probabilities differ after mapping, the implementation has changed more than the coordinates.