Information Matrices And Asymptotic Standard Errors

From Information to Parameter Uncertainty

An information matrix is useful not only for comparing designs but also for anticipating how precisely parameters can be estimated. A design with substantial information in every relevant parameter direction tends to produce smaller estimator variances. The inverse Fisher information gives a local, large-sample approximation to the covariance matrix of a maximum likelihood estimator (MLE). Its diagonal entries yield approximate variances; their square roots are asymptotic standard errors.

This connection is an approximation, not a promise that an estimator has an exactly normal distribution or that its finite-sample variance equals the inverse information. Its usefulness depends on regularity conditions and on the model and parameter values used to calculate the information. In nonlinear pharmacometric models, the approximation is often local: it describes uncertainty near a specified parameter value or fitted estimate.

Here \(I(\theta)\) denotes the information for all observations in the analysis. If independent observations contribute information additively, total information is the sum of their contributions. With \(n\) identically distributed independent observations, it is often \(n\) times the per-observation information. For an approximate design with per-subject information \(I(\xi;\theta)\), \(n\) subjects give total information \(nI(\xi;\theta)\), and the approximate covariance is \([nI(\xi;\theta)]^{-1}\). Be clear about whether a reported matrix is per subject or already total; otherwise the factor of \(n\) can be omitted or counted twice.

Why the Inverse Appears

The inverse-information result follows from a first-order approximation to the score equation. Suppose the true parameter \(\theta_0\) is an interior point, the model is correctly specified and identifiable, the likelihood is sufficiently smooth, and the observations provide increasing information. The MLE approximately solves \(U(\hat{\theta})=0\). A Taylor expansion around \(\theta_0\) gives

The derivative of the score is the Hessian of the log-likelihood. For a large sample it is approximately \(-I(\theta_0)\), while the score has mean zero and covariance \(I(\theta_0)\). Rearranging the expansion therefore gives

This justification also shows why the approximation is sensitive to assumptions: it relies on a well-behaved score, a locally quadratic log-likelihood, and an invertible information matrix. In applications, the unknown \(\theta_0\) is usually replaced by \(\hat{\theta}\), giving the estimated covariance approximation \(I(\hat{\theta})^{-1}\). With a design calculation, the matrix may instead be evaluated at a nominal parameter vector. The result is then a prediction of precision conditional on that model and parameter vector.

Standard Errors and Wald Confidence Intervals

For a \(p\)-dimensional parameter vector, let \(\widehat V=I(\hat{\theta})^{-1}\) be the estimated covariance approximation. The standard error for component \(\hat{\theta}_j\) is the square root of the \(j\)th diagonal element of \(\widehat V\). Off-diagonal elements describe approximate covariances, so parameter estimates can be strongly correlated even when each has a finite standard error. A diagonal entry of the information matrix itself is not, in general, the variance or the reciprocal variance for that component: the full matrix must be inverted when parameters are correlated.

For a scalar parameter, or one component of a parameter vector, an approximate two-sided \(100(1-\alpha)\%\) Wald confidence interval is

where \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. For a 95% interval, \(z_{0.975}\) is approximately 1.96. The interval is justified by the approximate normal sampling distribution of the estimator, not by a claim that the fixed parameter has a 95% probability of lying in this particular interval. In repeated comparable analyses, the procedure has approximately 95% coverage when its assumptions and approximation are adequate.

Reading a Multivariate Covariance Approximation

For a vector parameter, uncertainty in one component must be calculated after accounting for the other estimated parameters. The following hypothetical information matrix is a total information matrix for two parameters: \(\theta_1\), a clearance in L/h, and \(\theta_2\), a volume in L. Suppose the fitted values are \(2.00\) L/h and \(0.500\) L. Assume the likelihood is regular and locally well approximated by a quadratic around these interior estimates; the example illustrates the resulting Wald calculation, not evidence that the assumptions hold in a particular study.

For a linear contrast \(c^T\theta\), where \(c\) is a fixed vector, the approximate variance is \(c^T\widehat Vc\), and its standard error is the square root of that quantity. This is useful when the target is a combination of parameters rather than a single component. For a nonlinear transformation \(g(\theta)\), a first-order Taylor expansion gives the delta-method variance approximation

Alternatively, for a strictly increasing transformation of a scalar parameter, one can transform both endpoints of a confidence interval on the original parameter scale. The resulting interval need not be symmetric around the transformed estimate.

Assumptions, Diagnostics, and Alternatives

The inverse-information approximation is most defensible when the model is correctly specified, parameters are identifiable and away from problematic boundaries, the likelihood is smooth, and the data provide enough information for a local quadratic approximation. In a design calculation, these claims are conditional on the assumed observation model and nominal parameter values. A large number of observations alone does not resolve nonidentifiability or misspecification.

Inspect the fitted model and uncertainty summaries for warning signs. A singular or nearly singular information matrix, extremely large standard errors, or very strong parameter correlations may indicate that some parameter combinations are weakly informed. A Wald interval that extends far into implausible values, as in the volume example, suggests the symmetric approximation may be misleading. A strongly skewed likelihood, estimates near a boundary, or marked changes in standard errors across plausible parameter values also call for caution. These checks diagnose possible problems; none by itself proves that the model is invalid.

Expected Fisher information averages curvature over possible data under the model. Observed information, the negative Hessian of the log-likelihood at the actual data and fitted estimate, uses the curvature actually observed. Both can be used to form a local covariance approximation, but they can differ in finite samples. The choice should be stated, especially when reporting a matrix calculated by software or a design program.

Wald intervals are computationally simple and directly connected to the inverse information. Profile-likelihood intervals instead examine the likelihood over values of a target parameter while re-optimizing nuisance parameters; they can reflect asymmetry better, though they still depend on a suitable likelihood and can require more computation. A parametric bootstrap simulates data under a fitted model and refits it to approximate an estimator's sampling distribution; it can expose skewness or finite-sample behavior, but inherits the assumed model and may be unstable when refits fail. Robust sandwich covariance estimates can address certain forms of likelihood misspecification in large samples, but do not repair a poor mean model or eliminate weak identification. These alternatives are not automatic fixes; their assumptions and target of inference should be matched to the problem.

Information criteria such as A-, D-, and c-optimality summarize different aspects of the same information matrix, as covered in the earlier material on those criteria. Inference adds an important practical step: translate the matrix into standard errors, covariances, and intervals for specific targets, then judge whether the approximation is plausible on the parameter scale that matters. A design that is optimal under a criterion still relies on the model, parameter values, and asymptotic approximation used to calculate its information.