Turning the Hessian Into a Parameter Covariance Matrix

From Likelihood Curvature to Parameter Uncertainty

After fitting an NLME model, a point estimate alone does not describe how precisely the model parameters have been estimated. The curvature of the marginal likelihood near its maximum provides a local measure of that uncertainty. In particular, the inverse of the observed-information matrix estimates the covariance matrix of the fitted parameters. Its diagonal entries give estimated variances, while its off-diagonal entries describe how estimation errors in different parameters move together.

This is a local, model-based approximation, not a direct measurement of uncertainty and not the covariance matrix of the subject-level random effects. The calculation is meaningful only when the fitted solution and the likelihood surface meet suitable conditions. The earlier material on observed information from the marginal likelihood defines the curvature being used here; the focus now is its matrix inversion and interpretation.

The parameter vector may include typical values, covariate effects, random-effects variance or covariance parameters, and residual-error parameters, depending on how the model is parameterized. Keep the parameter ordering fixed: row \(i\) and column \(j\) of both matrices must refer to the same parameters. Each entry of \(\mathbf{J}\) has units reciprocal to the product of the units of its row and column parameters. Its inverse therefore has the units of a parameter product, as required for a covariance matrix.

Why the Inverse Appears

The inverse is justified by the local shape of the likelihood. At an interior maximum, the first derivative is zero. A second-order Taylor expansion around \(\hat{\boldsymbol{\theta}}\) gives, for parameter vectors close to the estimate,

Exponentiating this approximation gives a likelihood proportional to a multivariate normal-shaped kernel centered at \(\hat{\boldsymbol{\theta}}\). A normal density with covariance matrix \(\mathbf{C}\) has quadratic term \((\boldsymbol{\theta}-\hat{\boldsymbol{\theta}})^\mathsf{T}\mathbf{C}^{-1}(\boldsymbol{\theta}-\hat{\boldsymbol{\theta}})\). Matching the two quadratic terms yields \(\mathbf{C}=\mathbf{J}^{-1}\). This explains the curvature calculation; it does not make the likelihood exactly normal. The covariance interpretation is an asymptotic, local approximation under regularity conditions.

In practice, the approximation is most defensible when the marginal likelihood is sufficiently smooth near the estimate, the solution is a well-defined interior maximum, and the parameters are locally identifiable. The information must be positive definite for its ordinary inverse to be a positive-definite covariance estimate. A valid covariance matrix more generally need only be positive semidefinite: it can be singular and have zero eigenvalues or zero diagonal entries. That broader fact does not make a singular information matrix invertible.

Reading the Matrix and Checking Its Scale

For a covariance estimate \(\mathbf{C}=\mathbf{J}^{-1}\), each diagonal entry \(C_{ii}\) estimates the variance of \(\hat{\theta}_i\), and \(\sqrt{C_{ii}}\) is its standard error in the parameter's units. The correlation for parameters \(i\) and \(j\) is

For a valid covariance matrix, diagonal entries are nonnegative and correlations lie between \(-1\) and \(1\), when both variances are positive. A covariance matrix must be symmetric and positive semidefinite. If it is positive definite, it is also nonsingular; positive definiteness is required for the ordinary inverse-information estimate in this tutorial, not for every possible covariance matrix.

The values depend on the coordinates in which the Hessian was calculated. If an optimizer uses rescaled parameters, convert the covariance to the desired units before reporting standard errors. For a linear change of coordinates \(\boldsymbol{\theta}=\mathbf{A}\boldsymbol{\phi}+\mathbf{b}\), the covariance transforms exactly as \(\mathbf{C}_{\theta}=\mathbf{A}\mathbf{C}_{\phi}\mathbf{A}^\mathsf{T}\). This rule preserves both variances and covariances; scaling only the diagonal entries would generally be wrong.

When Inversion Fails or Misleads

Numerical optimizers and numerical Hessian calculations can produce a matrix that is singular, nearly singular, or not positive definite. These outcomes should be investigated rather than repaired mechanically. A nearly singular information matrix has a direction of low curvature: some combination of parameters can change substantially while the likelihood changes little. The resulting inverse may have very large variances and correlations near \(1\) or \(-1\). This often signals weak identifiability, excessive parameterization, sparse information in the design, or parameters that the data cannot distinguish well.

The following two-parameter example shows why a matrix that fails the positive-definiteness check cannot supply the intended covariance estimate.

A singular matrix deserves a separate interpretation. A matrix such as \(\begin{pmatrix}1&1\\1&1\end{pmatrix}\) is a valid positive-semidefinite covariance matrix: it represents two perfectly linearly dependent quantities with zero variance in their difference. But it has no ordinary inverse. Likewise, if the observed information is singular, its inverse cannot be used to obtain the ordinary covariance estimate. Generalized inverses or constraints may be useful in particular analyses, but they require an explicit model-specific interpretation and are not automatic substitutes for a failed inverse.

Before interpreting standard errors, inspect the fitted solution, the symmetry and eigenvalues of the information matrix, the magnitude of its condition number, and the resulting covariance and correlation matrices. Also compare the Hessian calculation across reasonable numerical settings if it was computed numerically. A positive-definite matrix is necessary for a nonsingular inverse-information covariance approximation, but it cannot by itself establish that the model is scientifically adequate or that the local approximation is accurate.

How This Approximation Compares With Alternatives

Observed information uses curvature from the actual fitted marginal log likelihood. Expected Fisher information instead uses the expectation of the information under the model; it describes average curvature across hypothetical data sets and may be more stable in some settings, but can be harder to calculate and need not match the observed curvature in a particular data set. Both rely on likelihood-based regularity and local asymptotic reasoning.

Profile likelihood and bootstrap approaches assess uncertainty through different calculations and can reveal asymmetry or behavior that a local quadratic approximation misses. They typically require more computation and their own assumptions and diagnostics. The inverse Hessian is attractive because it is readily available from many fitting procedures and summarizes joint local uncertainty in one matrix. It is least trustworthy when the likelihood is strongly nonquadratic, parameters are near boundaries, or the fit is weakly identified. It should be treated as one uncertainty assessment, not as a universal replacement for alternatives.