Asymptotic Covariance From Gaussian Information

From Gaussian Information to Estimator Uncertainty

A maximum-likelihood estimate is often reported as a vector, but uncertainty in its coordinates is not captured by a list of separate standard errors alone. The covariances between coordinates matter too: they describe how estimation errors tend to move together. Fisher information provides a principled route from a likelihood to this full covariance matrix.

The central large-sample result is that, under regularity conditions, the maximum-likelihood estimator is approximately multivariate normal. Its covariance is the inverse Fisher information for the full sample. For independent, identically distributed observations, the full-sample information is $n$ times the information per observation, so the covariance is approximately $I(\theta_0)^{-1}/n$. A differentiable transformation of the parameter has a corresponding covariance obtained by the delta method.

Definition: Let $Y_1,\ldots,Y_n$ be independent and identically distributed observations from a model indexed by a $p$-dimensional parameter $\theta$. For one observation, write $s(Y;\theta)=\partial\log f(Y;\theta)/\partial\theta$ for the score vector. The Fisher information per observation is $$ I(\theta)=E_\theta[s(Y;\theta)s(Y;\theta)^T]. $$ When the usual information identity applies, this also equals $-E_\theta[\partial^2\log f(Y;\theta)/\partial\theta\partial\theta^T]$. The Fisher information in the full sample is $nI(\theta)$.

The information identity requires suitable regularity, including conditions that permit differentiation under the integral defining the model’s total probability. In regular Gaussian models with parameter-independent support, these conditions hold. The matrix $I(\theta)$ must also be nonsingular to use its inverse in the asymptotic covariance formula.

The Multivariate MLE Result

The result below makes explicit the assumptions behind the information-based covariance. Let $\theta_0$ be the true parameter, and let $\hat\theta$ be a consistent MLE. Assume $\theta_0$ is an interior point of an open parameter space, the log-likelihood is twice continuously differentiable in a neighborhood of $\theta_0$, and the MLE satisfies the score equation with probability tending to one. Assume also that the score has mean zero, finite covariance $I(\theta_0)$, and satisfies the multivariate central limit theorem. Finally, suppose $I(\theta_0)$ is positive definite and a local uniform law of large numbers for the Hessian holds, with its expected value continuous at $\theta_0$ and equal to $-I(\theta_0)$ there.

Theorem: Under these conditions, $$ \sqrt{n}(\hat\theta-\theta_0)\xrightarrow{d}N_p\!\left(0,I(\theta_0)^{-1}\right). $$ Consequently, the asymptotic covariance of $\hat\theta$ is $I(\theta_0)^{-1}/n$. Here “asymptotic covariance” means the covariance matrix of the normal approximation to the estimator’s sampling distribution for large $n$; it need not equal its exact finite-sample covariance.

To prove the result, let $U_n(\theta)$ be the score vector for the full sample and $H_n(\theta)$ its Hessian matrix. The score is a sum of independent contributions, so the multivariate central limit theorem gives $U_n(\theta_0)/\sqrt n\xrightarrow{d}N_p(0,I(\theta_0))$. The score equation at the MLE is $U_n(\hat\theta)=0$, with probability tending to one.

For a vector-valued score, a scalar mean-value theorem does not in general supply one common intermediate point for every coordinate. Instead, apply the fundamental theorem of calculus to the score along the line segment from $\theta_0$ to $\hat\theta$. This gives the exact integral expansion $$ 0=U_n(\theta_0)+\left[\int_0^1 H_n\!\left(\theta_0+t(\hat\theta-\theta_0)\right)\,dt\right](\hat\theta-\theta_0). $$

Define the averaged Hessian divided by $n$ as $$ A_n=\int_0^1 \frac{H_n\!\left(\theta_0+t(\hat\theta-\theta_0)\right)}{n}\,dt. $$ Consistency implies that the entire segment in this integral approaches $\theta_0$ in probability. The assumed local uniform law of large numbers and continuity of the expected Hessian therefore imply $A_n\xrightarrow{p}-I(\theta_0)$. Since this limit is nonsingular, $A_n^{-1}\xrightarrow{p}-I(\theta_0)^{-1}$. Rearranging the integral expansion, $$ \sqrt n(\hat\theta-\theta_0)=-A_n^{-1}\frac{U_n(\theta_0)}{\sqrt n}. $$ Slutsky’s theorem now gives the stated normal limit: the factor $-A_n^{-1}$ converges in probability to $I(\theta_0)^{-1}$, and the score term converges in distribution to $N_p(0,I(\theta_0))$. The resulting covariance is $I(\theta_0)^{-1}I(\theta_0)I(\theta_0)^{-1}=I(\theta_0)^{-1}$. $\square$

Worked Covariance Calculations

Worked Example: A Gaussian Mean With Known Covariance

Suppose $Y_1,\ldots,Y_n$ are independent $p$-dimensional observations from $N_p(\mu,\Sigma)$, where $\Sigma$ is known and positive definite. Find the asymptotic covariance of the MLE of $\mu$.

For one observation, the log-likelihood, omitting a term constant in $\mu$, is $-\tfrac12(Y-\mu)^T\Sigma^{-1}(Y-\mu)$. Differentiating with respect to $\mu$ gives the score $s(Y;\mu)=\Sigma^{-1}(Y-\mu)$. Its information is $$ I(\mu)=E_\mu[s(Y;\mu)s(Y;\mu)^T] =\Sigma^{-1}E_\mu[(Y-\mu)(Y-\mu)^T]\Sigma^{-1} =\Sigma^{-1}\Sigma\Sigma^{-1} =\Sigma^{-1}. $$

The MLE is $\hat\mu=\bar Y$. Thus its asymptotic covariance is $\Sigma/n$. In this particular model, this is also its exact covariance: since the sample mean of independent Gaussian vectors is $N_p(\mu,\Sigma/n)$, its sampling covariance is exactly $\Sigma/n$ for every $n$. This exact agreement is a feature of this model, not a general consequence of the asymptotic theorem.

Worked Example: Estimating a Gaussian Mean and Variance

Let $Y_1,\ldots,Y_n$ be independent $N(\mu,v)$ observations, with both $\mu$ and $v>0$ unknown. Use the parameter order $\theta=(\mu,v)^T$ to find the asymptotic covariance of the MLE.

For one observation $Y$, the score components are $$ s_\mu(Y;\theta)=\frac{Y-\mu}{v}, \qquad s_v(Y;\theta)=-\frac{1}{2v}+\frac{(Y-\mu)^2}{2v^2}. $$ The centered Gaussian has $E[(Y-\mu)^2]=v$, $E[(Y-\mu)^3]=0$, and $E[(Y-\mu)^4]=3v^2$. Therefore $E[s_\mu^2]=1/v$, $E[s_\mu s_v]=0$, and $$ E[s_v^2] =\frac{E[((Y-\mu)^2-v)^2]}{4v^4} =\frac{3v^2-2v^2+v^2}{4v^4} =\frac{1}{2v^2}. $$ Thus $I(\theta)=\operatorname{diag}(1/v,1/(2v^2))$, and $I(\theta)^{-1}=\operatorname{diag}(v,2v^2)$. The MLEs are $\hat\mu=\bar Y$ and $\hat v=n^{-1}\sum_{i=1}^n(Y_i-\bar Y)^2$; the theorem gives asymptotic covariance $\operatorname{diag}(v/n,2v^2/n)$.

For example, at $\mu=3$, $v=4$, and $n=200$, this matrix is $$ \begin{pmatrix}4/200&0\\0&2(4^2)/200\end{pmatrix} = \begin{pmatrix}0.02&0\\0&0.16\end{pmatrix}. $$ The corresponding asymptotic standard errors are $\sqrt{0.02}\approx0.1414$ for the mean and $\sqrt{0.16}=0.4$ for the variance. The zero off-diagonal entry describes the limiting covariance; it does not by itself assert independence of the finite-sample MLEs.

Nonlinear Parameters and the Delta Method

Often the parameter of interest is $g(\theta)$, not $\theta$ itself. Suppose $g$ maps an open neighborhood of $\theta_0$ into $\mathbb R^q$ and is differentiable at $\theta_0$. Write $J_g(\theta_0)$ for its $q$-by-$p$ Jacobian matrix, whose entry in row $r$, column $j$ is $\partial g_r/\partial\theta_j$.

Theorem: If $\sqrt n(\hat\theta-\theta_0)\xrightarrow{d}N_p(0,I(\theta_0)^{-1})$ and $g$ is differentiable at $\theta_0$, then $$ \sqrt n\{g(\hat\theta)-g(\theta_0)\}\xrightarrow{d} N_q\!\left(0,J_g(\theta_0)I(\theta_0)^{-1}J_g(\theta_0)^T\right). $$ Thus the asymptotic covariance of $g(\hat\theta)$ is $J_g(\theta_0)I(\theta_0)^{-1}J_g(\theta_0)^T/n$.

Differentiability gives the local expansion $g(\hat\theta)-g(\theta_0)=J_g(\theta_0)(\hat\theta-\theta_0)+r_n$, where $\|r_n\|/\|\hat\theta-\theta_0\|\xrightarrow{p}0$ whenever $\hat\theta\ne\theta_0$. The MLE limit implies $\hat\theta-\theta_0=O_p(n^{-1/2})$, so $\sqrt n r_n=o_p(1)$. Multiplication of the limiting normal vector by $J_g(\theta_0)$, followed by Slutsky’s theorem, proves the result. $\square$

Worked Example: A Nonlinear Transformation of Two Gaussian Means

For each subject, observe independent $X\sim N(a,4)$ and $Y\sim N(b,32)$, with known variances and unknown means $a,b$. At the true parameter $(a,b)=(1,2)$, find the asymptotic covariance of the estimate of $g(a,b)=(a/2+(a-1)^2-b/8,\ b/4)^T$ when $n=100$ independent subjects are observed.

The per-subject information for $(a,b)^T$ is $I=\operatorname{diag}(1/4,1/32)$, so $I^{-1}=\operatorname{diag}(4,32)$. The Jacobian is $$ J_g(a,b)= \begin{pmatrix} 1/2+2(a-1)&-1/8\\ 0&1/4 \end{pmatrix}, \qquad J_g(1,2)= \begin{pmatrix}1/2&-1/8\\0&1/4\end{pmatrix}. $$ The first coordinate of $g$ is nonlinear, but its derivative at the true parameter is $1/2$ with respect to $a$. Multiplying in the required order, $$ J_g I^{-1} = \begin{pmatrix}1/2&-1/8\\0&1/4\end{pmatrix} \begin{pmatrix}4&0\\0&32\end{pmatrix} = \begin{pmatrix}2&-4\\0&8\end{pmatrix}, $$ and hence $$ J_g I^{-1}J_g^T = \begin{pmatrix}2&-4\\0&8\end{pmatrix} \begin{pmatrix}1/2&0\\-1/8&1/4\end{pmatrix} = \begin{pmatrix}3/2&-1\\-1&2\end{pmatrix}. $$ After dividing by $n=100$, the transformed estimator has asymptotic covariance $$ \begin{pmatrix}0.015&-0.01\\-0.01&0.02\end{pmatrix}. $$ The negative off-diagonal entry means the two transformed estimation errors tend to vary in opposite directions under this normal approximation. The transformed parameter value itself is $g(1,2)=(0.25,0.5)^T$.

Using and Interpreting the Covariance

The matrix formula provides more than marginal standard errors. Its diagonal entries give the asymptotic variances of the coordinates, while its off-diagonal entries describe their asymptotic covariances. For a scalar contrast $c^T\theta$, the gradient is $c^T$, so its asymptotic variance is $c^TI(\theta_0)^{-1}c/n$. For a nonlinear vector quantity, the Jacobian carries the local effect of parameter estimation into every transformed coordinate.

In applications, $\theta_0$ is unknown. A common estimated covariance substitutes $\hat\theta$ into the per-observation information matrix and uses $I(\hat\theta)^{-1}/n$, or substitutes it into the transformed version as well. This plug-in estimate is justified when the information varies continuously near $\theta_0$, $\hat\theta\xrightarrow{p}\theta_0$, and the inverse remains well behaved. It is an estimate of the asymptotic covariance, not a claim that the estimator’s exact sampling law is normal.

Common Pitfall

Forgetting the sample-size factor. If $I(\theta)$ denotes information per observation, the covariance approximation is $I(\theta)^{-1}/n$, not $I(\theta)^{-1}$. If using full-sample information $nI(\theta)$, its inverse already includes the factor $1/n$.
Applying the transformation to the covariance incorrectly. For $g(\theta)$, use $J_g I^{-1}J_g^T/n$. Do not simply invert the Jacobian, or transform each diagonal variance separately while discarding covariance terms.
Treating an asymptotic covariance as an exact one. The normal limit describes large-sample behavior. Exact finite-sample covariance must be established from the model itself, as it was separately verified for the Gaussian sample mean above.

Key takeaway: Under regularity conditions, the MLE’s large-sample covariance is the inverse per-observation Fisher information divided by $n$. A differentiable transformation carries that covariance forward through its Jacobian on both sides.

Check Your Understanding

Use the information and transformation formulas to answer these questions.

If the information per observation is $I(\theta_0)$, what is the asymptotic covariance of the MLE based on $n$ independent observations?
Why does the score expansion use an integral of Hessian matrices rather than assuming one common intermediate point for every score coordinate?
For a differentiable scalar function $g(\theta)$, what Jacobian-based expression gives the asymptotic variance of $g(\hat\theta)$?
In the two-mean example, why does the transformed covariance have a nonzero off-diagonal entry even though the original information matrix is diagonal?
What additional fact is needed before calling an asymptotic covariance matrix the exact finite-sample covariance?