Gaussian Scores and Fisher Information

From Gaussian Likelihood to Score

Likelihood-based inference depends on how the log-likelihood changes as its parameters move. Its first derivative, the score, records the local direction and rate of change; the score's covariance gives Fisher information, which measures the precision available in the model. In a Gaussian model, these quantities can be derived explicitly for both the mean and covariance parameters.

Let \(X_1,\ldots,X_n\) be independent observations from \(N_p(\mu,\Sigma)\), where \(\mu\in\mathbb R^p\) and \(\Sigma\) is a positive-definite \(p\times p\) covariance matrix. Write \(y_i=X_i-\mu\). The log-likelihood, up to no omitted parameter-dependent terms, is

The score for a vector parameter is its column vector of partial derivatives. Because \(\Sigma\) is symmetric, its derivatives are most cleanly specified by symmetric directions: a perturbation \(H\) is any symmetric matrix, and the directional derivative is the derivative along \(\Sigma+tH\) at \(t=0\). This avoids treating the two copies of each off-diagonal covariance entry as separate parameters.

Mean and Covariance Scores

Differentiating the quadratic term with respect to \(\mu\) gives the mean score. For the covariance score, use the differential identities \(d\log|\Sigma|=\operatorname{tr}(\Sigma^{-1}d\Sigma)\) and \(d\Sigma^{-1}=-\Sigma^{-1}(d\Sigma)\Sigma^{-1}\). Both apply because \(\Sigma\) is positive definite.

For a symmetric covariance perturbation \(H\), the directional score is

Equivalently, under the trace pairing \(\operatorname{tr}(GH)\), the symmetric matrix representing the covariance score is

To check the covariance derivative, the log-determinant contributes \(-\tfrac n2\operatorname{tr}(\Sigma^{-1}H)\). The quadratic term contributes \(\tfrac12\sum_i y_i^\mathsf{T}\Sigma^{-1}H\Sigma^{-1}y_i\), which equals \(\tfrac12\sum_i\operatorname{tr}(\Sigma^{-1}y_i y_i^\mathsf{T}\Sigma^{-1}H)\). Adding the terms gives the stated directional score.

Fisher Information for Mean and Covariance

The covariance information is naturally a bilinear form on symmetric directions. For directions \(H\) and \(K\), its value is the expected product of the two directional scores. The next result gives all blocks of the information for the Gaussian model.

If the covariance is represented by its \(p(p+1)/2\) distinct entries, this formula gives the information matrix in those coordinates. For example, the coordinate for a diagonal entry uses direction \(E_{jj}\), while an off-diagonal coordinate \(\sigma_{jk}\), \(j<k\), uses \(E_{jk}+E_{kj}\).

Proof. Set \(y_i=X_i-\mu\). The mean score is \(\Sigma^{-1}\sum_i y_i\). Independence and \(E[y_i]=0\), \(\operatorname{Var}(y_i)=\Sigma\) give

For one observation, its covariance directional score is \(\tfrac12\{y^\mathsf{T}Ay-\operatorname{tr}(A\Sigma)\}\), where \(A=\Sigma^{-1}H\Sigma^{-1}\). Likewise, direction \(K\) uses \(B=\Sigma^{-1}K\Sigma^{-1}\). Write \(y=\Sigma^{1/2}z\), where \(z\sim N_p(0,I_p)\), and set \(C=\Sigma^{1/2}A\Sigma^{1/2}\), \(D=\Sigma^{1/2}B\Sigma^{1/2}\). Then the centered quadratic terms are \(z^\mathsf{T}Cz-\operatorname{tr}(C)\) and \(z^\mathsf{T}Dz-\operatorname{tr}(D)\).

For independent standard normal coordinates, \(E[z_a z_b z_c z_d]\) is the sum of the three products of Kronecker deltas corresponding to the pairings \((a,b)(c,d)\), \((a,c)(b,d)\), and \((a,d)(b,c)\). This follows from independence: a product has expectation zero if an index occurs an odd number of times; if all indices occur in pairs, the needed moments are \(E[z_j^2]=1\) and \(E[z_j^4]=3\). Expanding the two quadratic forms with this identity yields

Thus the covariance-direction information from one observation is \(\tfrac12\operatorname{tr}(CD)=\tfrac12\operatorname{tr}(\Sigma^{-1}H\Sigma^{-1}K)\); independence makes the \(n\)-observation information \(n\) times this value. Finally, the product of a mean score and a covariance score has expectation zero: after centering, each term is a product of three components of a centered Gaussian vector, and every such third moment is zero. Cross-observation terms vanish by independence and zero means. This proves all stated blocks. \(\square\)

The information is therefore block diagonal between mean and covariance under this parameterization, but covariance entries need not be mutually orthogonal. The trace formula handles their relationships without requiring a long coordinate-by-coordinate expansion.

Information Identities in a Scalar Gaussian Model

For a scalar parameter, two common expressions for information are the variance of the score and the negative expected second derivative of the log-likelihood. In regular models they agree; the Gaussian example makes the calculation and the distinction from sample-specific curvature explicit. Let \(X_1,\ldots,X_n\) be independent \(N(\mu,v)\) observations, where \(v>0\), and write \(S_2=\sum_{i=1}^n(X_i-\mu)^2\). The log-likelihood is

Why the Distinction Matters

The score identifies how the likelihood responds locally to a parameter change, while Fisher information summarizes the score's variability under the model. The mean-covariance cross block being zero is a precise statement about this Gaussian parameterization; it does not say that sample estimates are independent in every finite-sample setting or that all covariance coordinates are uncorrelated.

A frequent error is to call the negative observed Hessian “the Fisher information” without taking an expectation. In the numerical calculation above, the observed negative curvature for \(v\) is \(1/8\), while the Fisher information is \(1/2\). The correction is to distinguish the data-dependent Hessian from its model expectation. A second common error is to differentiate with respect to one off-diagonal matrix entry while forgetting that a symmetric covariance perturbation changes both entries; using the direction \(E_{jk}+E_{kj}\) makes the coordinate derivative unambiguous.