Estimating Parameter Uncertainty From the Covariance Matrix

What the Fixed-Effects Covariance Matrix Describes

An NLME fit produces estimates of population fixed effects, such as typical parameter values and covariate effects. Those estimates are uncertain because they were obtained from finite, variable data. The estimated covariance matrix of the fixed-effects estimates summarizes that uncertainty locally: its diagonal entries describe the estimated variances of individual estimates, while its off-diagonal entries describe how pairs of estimates vary together.

This matrix is not the random-effects covariance matrix. The latter describes between-subject variation in individual parameters. The fixed-effects covariance matrix instead describes uncertainty in the estimated population parameters. Keeping these two sources separate matters when assessing how much confidence to place in a fitted value or derived quantity.

For a single estimate, the estimated standard error is \(SE(\hat{\theta}_j)=\sqrt{V_{jj}}\). The covariance is not itself a standard error: it has squared parameter units on the diagonal, and the units of the two parameters combined off the diagonal. A correlation can be calculated as \(V_{jk}/\sqrt{V_{jj}V_{kk}}\), which puts the association between estimates on a scale from \(-1\) to \(1\).

The covariance matrix is commonly obtained from the curvature of the fitted likelihood, or an equivalent estimation criterion, near its optimum. It therefore supports a local uncertainty approximation. It does not by itself establish that the fitted model is correct, that the likelihood is well behaved far from the optimum, or that a normal approximation is adequate.

Why the Normal Approximation Appears

Under regularity conditions, an estimator based on a sufficiently informative sample is approximately multivariate normal near the true parameter. In practical terms, this approximation is most credible when the parameters are identifiable, the optimum is well inside the allowed parameter space, the likelihood is reasonably quadratic near the optimum, and the covariance estimate is stable. In that setting, a marginal estimate can be summarized approximately as \(\hat{\theta}_j\) plus or minus a normal quantile times its standard error.

The local justification comes from approximating the log likelihood around its maximum. Let \(\ell(\theta)\) be the log likelihood and let \(\hat{\theta}\) maximize it. A second-order Taylor expansion around \(\hat{\theta}\), where the first derivative is zero, gives

where \(J(\hat{\theta})\) is the observed information matrix, the negative Hessian of the log likelihood at the estimate. If this curvature matrix is nonsingular and the quadratic approximation is adequate nearby, the likelihood has an approximately normal shape, with covariance approximately \(J(\hat{\theta})^{-1}\). An estimator's covariance matrix may be calculated using this or a related curvature-based method. The argument explains both the appeal and the limitation: the approximation describes local curvature, not the full likelihood shape.

For a marginal 95% Wald interval, the usual calculation is \(\hat{\theta}_j\pm1.96SE(\hat{\theta}_j)\). Its frequentist interpretation is that, under the model and repeated-sampling conditions supporting the approximation, intervals constructed this way would cover the fixed parameter about 95% of the time. It is not a statement that the particular interval has a 95% probability of containing the parameter.

Uncertainty for a Linear Combination

Many quantities of interest combine several fixed effects. For a fixed vector \(a\), the linear combination is \(a^\mathsf{T}\theta\), estimated by \(a^\mathsf{T}\hat{\theta}\). Its estimated variance is not generally the sum of marginal variances alone. It is

This expression includes the covariance terms. If two estimates move together, their covariance increases the uncertainty of some combinations and decreases the uncertainty of others. Omitting the off-diagonal terms can therefore produce an incorrect standard error even when every marginal standard error is correct.

The variance formula follows directly from the definition of covariance. Write \(a^\mathsf{T}\hat{\theta}=\sum_j a_j\hat{\theta}_j\). Expanding the variance of this sum gives \(\sum_j a_j^2\operatorname{Var}(\hat{\theta}_j)+2\sum_{j<k}a_ja_k\operatorname{Cov}(\hat{\theta}_j,\hat{\theta}_k)\), which is exactly the matrix product \(a^\mathsf{T}Va\). This also shows why every covariance term must be retained.

When Is a Normal Approximation Plausible?

A covariance matrix gives a scale for local uncertainty, but it does not diagnose all aspects of the likelihood. Plausibility should be judged using the estimation context as well as the numerical matrix. Useful checks include:

• Interior estimates: the estimate should not sit at, or very close to, a parameter boundary. A symmetric normal approximation can assign substantial mass to impossible values near a boundary.
• Identifiability and curvature: estimates should not be supported by an almost flat likelihood direction. Very large standard errors, near-singular covariance matrices, or strong parameter correlations can signal weak identification.
• Optimization and matrix validity: the fit should have converged appropriately, and the covariance matrix should be symmetric and positive semidefinite up to numerical tolerance. A materially negative diagonal variance is not interpretable as uncertainty and requires investigation.
• Likelihood shape: strong skewness or asymmetry around the optimum weakens a symmetric Wald interval, even if the optimizer reports a covariance matrix.
• Scale and parameterization: interpret standard errors on the scale used for estimation. A normal approximation on one scale need not remain normal after a nonlinear transformation.

These checks are warnings and evidence, not a mechanical pass/fail test. A positive-definite matrix is necessary for a nondegenerate multivariate normal approximation, but it does not prove that the likelihood is quadratic or that the model is well specified. Conversely, high correlation alone does not establish failure; it may reflect a real trade-off between parameters, although a value near \(1\) or \(-1\) can indicate that the data distinguish their effects poorly.

Choosing and Interpreting an Uncertainty Summary

Covariance-based standard errors are compact and efficient when the likelihood is approximately quadratic and the fit is well identified. They provide marginal summaries and uncertainty for linear combinations without requiring a new model fit. The principal alternative summaries answer the same broad uncertainty question in different ways: profile-likelihood intervals use likelihood behavior away from the optimum and can reveal asymmetry; bootstrap intervals use variation across resampled data sets; Bayesian posterior summaries describe uncertainty conditional on a prior and likelihood. These approaches have their own assumptions and costs. They are not interchangeable by default, and none repairs a misspecified model automatically.

A common mistake is to treat the reported standard errors as independent simply because they are listed separately. For example, calculating uncertainty for \(\theta_1+\theta_2\) using only \(V_{11}+V_{22}\) silently assumes the covariance is zero. The correction is to use the complete matrix and calculate \(a^\mathsf{T}Va\). Another mistake is to interpret a narrow covariance-based interval as proof of certainty: it quantifies local estimation uncertainty under the fitted model and approximation, not uncertainty from every modeling choice or source of variability.