Covariance Step Settings and Uncertainty Estimates

Why Covariance-Step Settings Matter

A fitted model gives parameter estimates, but those estimates alone do not show how precisely the data determine each parameter. A covariance step uses local curvature around the fitted optimum to estimate uncertainty: its diagonal entries yield standard errors, while its off-diagonal entries describe how estimation errors in different parameters vary together. The options chosen for that calculation can therefore affect the reported uncertainty without changing the fitted parameter values or objective function.

The earlier material on Hessian-based standard errors at the optimum introduced the inverse-curvature calculation. Here the emphasis is on what a covariance step has to choose: which curvature to estimate, how to calculate derivatives, which parameter scale to use, and whether to use a model-based or robust uncertainty estimate. These choices matter most when curvature is estimated numerically, the model is nonlinear, or the likelihood assumptions are imperfect.

This convention uses the negative log likelihood. If software instead minimizes \(Q(\theta)=-2\ell(\theta)\), then \(\nabla^2 Q=2H\), so the corresponding covariance estimate is \(2\{\nabla^2 Q(\hat{\theta})\}^{-1}\). Check the objective-function convention before comparing reported standard errors; omitting this factor would make them too small by a factor of \(\sqrt{2}\).

What a Covariance Step Calculates

A covariance step is a post-estimation calculation centered on the fitted parameter vector. It estimates local curvature, inverts the curvature matrix when appropriate, and converts the resulting covariance matrix into standard errors and correlations. Its result is conditional on the fitted optimum and on the model and data used for estimation. It is not a new parameter-estimation step.

For a covariance matrix \(\widehat{\Sigma}\), the standard error of parameter \(j\) is \(\sqrt{\widehat{\Sigma}_{jj}}\). The estimated correlation between parameters \(j\) and \(k\) is \(\widehat{\Sigma}_{jk}/\sqrt{\widehat{\Sigma}_{jj}\widehat{\Sigma}_{kk}}\), provided both estimated variances are positive. Correlation is dimensionless, whereas standard errors have the same units as their parameters. A covariance matrix must be symmetric and positive semidefinite; it need not be positive definite. A singular matrix can still be a valid covariance matrix, although it cannot provide a nonsingular full-rank inverse-information estimate.

The calculation depends on settings that are sometimes grouped under a software label such as “covariance step.” Important choices include whether the curvature is observed or expected information, how derivatives are calculated, and whether uncertainty is model-based or uses a sandwich estimator. Software may also offer controls for numerical precision or derivative step sizes. Such controls affect how curvature is approximated; they do not make a poor local quadratic approximation accurate.

Why Inverse Curvature Represents Uncertainty

The inverse-Hessian result follows from a local approximation. At an interior optimum, the gradient of \(L\) is zero. For a small change \(d=\theta-\hat{\theta}\), a second-order Taylor expansion gives

Because the likelihood is proportional to \(\exp\{-L(\theta)\}\), this approximation gives a local likelihood proportional to \(\exp\{-\frac{1}{2}d^\mathsf{T}H d\}\). A multivariate normal density has this quadratic form when its covariance matrix is \(H^{-1}\). Thus the inverse curvature estimates the covariance of the parameter estimates when the local approximation and likelihood-based assumptions are suitable. This is an approximation, not a guarantee that the sampling distribution is exactly normal.

The argument requires a well-defined interior optimum, sufficiently smooth likelihood near that optimum, and informative curvature in the parameter directions being estimated. For the ordinary full-rank inverse-information estimate, \(H\) must be nonsingular; positive curvature in all directions is expected at a well-behaved local minimum of \(L\). Boundary estimates, weakly informed directions, or a poor local approximation can make the resulting standard errors misleading even if software returns numbers.

Alternative Curvature and Uncertainty Calculations

The observed information is the curvature of the realized negative log likelihood at the estimate. Expected Fisher information instead averages the information over data generated by the fitted model. In regular, well-specified settings and with sufficient information, these approaches are often similar, but they are not generally numerically identical in a finite sample. Expected information may be analytically available or calculated by averaging over the model; observed information uses the curvature at the data actually observed.

A separate choice is how to obtain derivatives. Analytic second derivatives, when available and correctly implemented, avoid finite-difference approximation error. Numerical differentiation estimates curvature using changes in the objective at nearby parameter values; its result can depend on step size and numerical precision. Two derivative methods aim at the same observed Hessian, but may yield different estimates because their approximations differ. A small numerical discrepancy is not itself evidence that either result is correct; compare methods on the same objective and scale and check that estimates are stable to reasonable computational settings.

A sandwich estimate addresses a different issue: it replaces the model-based variance calculation with an estimate designed to remain valid under some forms of likelihood misspecification. For independent observational units, let \(s_i\) be the score contribution from unit \(i\), the derivative of its log likelihood. Let \(B=\sum_i s_i s_i^\mathsf{T}\), and let \(H\) be the observed Hessian of the total negative log likelihood. The sandwich estimate is \(H^{-1}BH^{-1}\). If observations are clustered, the score contributions used for \(B\) should be summed within independent clusters before forming the outer products. The sandwich approach relies on suitable independence, regularity, and asymptotic conditions; it does not repair a structurally inadequate model or weak information in the data.

Parameter Scales, Diagnostics, and Limitations

Curvature depends on the parameter scale. If \(z=g(\theta)\) is a smooth reparameterization, covariance estimates on the new scale should be transformed back using the Jacobian. Locally, if \(\theta=h(z)\) and \(J=\partial\theta/\partial z^\mathsf{T}\), then \(\widehat{\Sigma}_\theta\) is approximately \(J\widehat{\Sigma}_zJ^\mathsf{T}\). This matters when estimation uses transformed parameters but results are reported in the original units. A covariance matrix on a log scale is not itself a covariance matrix in the original units.

Before interpreting a covariance-step result, check that it corresponds to the intended optimum and objective convention, that the reported parameter scale is clear, and that the matrix is symmetric and positive semidefinite up to numerical rounding. Check that diagonal entries used for standard errors are nonnegative and that correlations lie between \(-1\) and \(1\), allowing for small rounding error. A symmetric positive semidefinite matrix may be singular; singularity means a full-rank inverse estimate is unavailable, not automatically that the matrix is an invalid covariance matrix. A substantially negative eigenvalue or a correlation outside the valid range indicates an invalid or numerically unreliable result that should not be reported as an ordinary covariance estimate.

Also compare uncertainty estimates across reasonable derivative calculations or information choices when appropriate. Large changes can indicate sensitivity to numerical approximation or to the information assumption. Numerical stability is necessary for confidence in the calculation, but it is not sufficient: a stable local estimate can still be misleading near a boundary or when the likelihood is poorly approximated by a quadratic. These standard errors are local, asymptotic summaries, not guaranteed measures of uncertainty in every finite sample.

Common Pitfall

A common mistake is to say that a covariance matrix must be positive definite, and therefore any singular covariance matrix is invalid. The correction is that covariance matrices must be symmetric and positive semidefinite; positive definiteness is an additional full-rank property. For example, if two random quantities are identical and each has variance \(1\), their covariance matrix is \(\begin{pmatrix}1&1\\1&1\end{pmatrix}\), which is valid but singular. In an inverse-information calculation, singularity prevents an ordinary nonsingular inverse and signals that the calculation does not yield a full-rank estimate; it does not by itself prove that every covariance matrix with a zero eigenvalue is invalid.

A second pitfall is to treat different standard errors as evidence that the parameter estimates changed. Observed, expected, numerical, and sandwich calculations can produce different uncertainty estimates around the same fitted parameters. Report which calculation produced the uncertainty and why it is appropriate; do not silently select the smallest standard error.