Estimating Standard Errors From Information

From Information to Standard Errors

A maximum-likelihood estimate is a point estimate, but actuarial decisions often also require an assessment of its precision. Fisher information provides a model-based way to estimate that precision. For one parameter, the calculation uses the reciprocal of information; for several parameters, it uses the inverse of an information matrix. The distinction between information per observation and information for the full sample is especially important: using the wrong one changes the resulting standard error by a factor involving the square root of the sample size.

Definition: Let $\hat{\theta}$ be a maximum-likelihood estimate based on $n$ independent observations. The expected information for the sample is $I_n(\theta)=E_\theta[-\ell''(\theta)]$, where $\ell$ is the sample log-likelihood. The observed information is $J_n(\theta)=-\ell''(\theta)$, calculated from the observed data. In practice, estimate either information quantity by evaluating it at $\hat{\theta}$. The estimated standard error is the square root of the resulting estimated variance.

For a vector parameter $\boldsymbol{\theta}$, the second derivative is replaced by the Hessian matrix of second partial derivatives. The expected information matrix is $I_n(\boldsymbol{\theta})=E_{\boldsymbol{\theta}}[-\nabla^2\ell(\boldsymbol{\theta})]$, and the observed information matrix is $J_n(\boldsymbol{\theta})=-\nabla^2\ell(\boldsymbol{\theta})$. Both matrices describe curvature of the log-likelihood: greater curvature generally corresponds to greater precision.

Theorem: Under the regularity conditions for maximum-likelihood asymptotic normality, an interior, identifiable MLE has estimated covariance matrix $$ \widehat{\operatorname{Cov}}(\hat{\boldsymbol{\theta}})\ \approx\ I_n(\hat{\boldsymbol{\theta}})^{-1} \qquad\text{or}\qquad \widehat{\operatorname{Cov}}(\hat{\boldsymbol{\theta}})\ \approx\ J_n(\hat{\boldsymbol{\theta}})^{-1}. $$ For a scalar parameter, this reduces to $\widehat{\operatorname{Var}}(\hat{\theta})\approx 1/I_n(\hat{\theta})$ or $1/J_n(\hat{\theta})$, and the estimated standard error is the square root of that quantity. For a vector parameter, the standard error of component $j$ is the square root of diagonal entry $j,j$ of the inverse matrix.

These are large-sample, model-based approximations, not exact finite-sample variances. Their use requires suitable regularity, such as a correctly specified model, independent observations, an identifiable parameter, an interior MLE, and finite, nonsingular information. If the fitted information matrix is singular or nearly singular, it cannot provide reliable standard errors by ordinary inversion. The inverse information is for the parameterization used in the likelihood; its entries and standard errors are expressed in that parameterization's units.

Why Inverting Information Gives an Approximate Covariance

The result follows from the asymptotic normality of the MLE. Write $I_1(\boldsymbol{\theta})$ for expected information per observation. For independent, identically distributed observations, total expected information is $I_n(\boldsymbol{\theta})=nI_1(\boldsymbol{\theta})$. Asymptotically, the estimation error has covariance approximately $I_1(\boldsymbol{\theta})^{-1}/n$. Since $$ \bigl(nI_1(\boldsymbol{\theta})\bigr)^{-1} =\frac{1}{n}I_1(\boldsymbol{\theta})^{-1}, $$ this is exactly the inverse of the total information. Replacing the unknown parameter by $\hat{\boldsymbol{\theta}}$ gives the estimated covariance. The same reasoning motivates observed information: it uses the curvature in the actual sample rather than its model expectation.

For a matrix, the diagonal entries of the inverse are estimated variances, and the off-diagonal entries are estimated covariances. In general, it is not correct to invert only the diagonal entries of the information matrix: parameter dependence is represented by its off-diagonal entries, and these affect the inverse.

Worked Example: Standard Error of an Exponential Rate Estimate

A hypothetical portfolio has 25 independent, uncensored claim severities $X_1,\ldots,X_{25}$, each modeled as exponential with rate $\lambda>0$, so $f_X(x)=\lambda e^{-\lambda x}$ for $x\geq 0$. The observed total severity is $\sum x_i=50$. Estimate the rate's standard error using information.

Method and estimate. The sample log-likelihood is $\ell(\lambda)=25\log\lambda-\lambda\sum x_i$. Setting its derivative equal to zero gives $25/\lambda-\sum x_i=0$, so $\hat{\lambda}=25/50=0.5$.

Expected information. The second derivative is $\ell''(\lambda)=-25/\lambda^2$. Thus the sample information is $I_{25}(\lambda)=25/\lambda^2$, and at the fitted rate, $I_{25}(\hat{\lambda})=25/(0.5)^2=25/0.25=100$.

Standard error. The estimated variance is $1/100=0.01$, so the estimated standard error is $\sqrt{0.01}=0.10$ rate units. The observed information has the same expression here, so it gives the same result.

Check and trap avoided. Information is for all 25 observations, not one observation. Using per-observation information $1/(0.5)^2=4$ as though it were total information would give the incorrect standard error $\sqrt{1/4}=0.50$. Multiplying by the sample size gives $100$ and the correct standard error $0.10$.

Scalar and Multivariate Calculations

A useful calculation sequence is to write the likelihood and parameterization, find the MLE, form expected or observed information for the whole sample, evaluate it at the MLE, and then take a reciprocal or matrix inverse. Only after inversion should diagonal entries be square-rooted to obtain component standard errors.

Find the fitted parameters.
Use the likelihood and the data to obtain $\hat{\boldsymbol{\theta}}$.

Construct sample information.
Use the negative expected Hessian or the negative observed Hessian. Check whether a formula is per observation or for the full sample.

Evaluate and invert.
Substitute the fitted parameter values, then invert the scalar or matrix information.

Extract standard errors.
Take the square root of a scalar variance or of each diagonal covariance entry. Retain off-diagonal entries when covariance information is needed.

Worked Example: Two-Parameter Normal Model

Suppose five hypothetical independent claim severities follow a normal distribution $X_i\sim N(\mu,\sigma^2)$, with both $\mu$ and $q=\sigma^2$ unknown. The observations are $1,2,3,4,5$. Estimate the standard errors of the MLEs of $\mu$ and $q$, using the parameter vector $(\mu,q)$.

Method and fitted values. The sample mean is $\bar{x}=(1+2+3+4+5)/5=3$. The squared deviations from 3 are $4,1,0,1,4$, with sum $10$. The normal MLE for the variance uses divisor $n$, so $\hat{\mu}=3$ and $\hat{q}=10/5=2$.

Expected information. In the parameterization $(\mu,q)$, the expected information matrix for $n$ observations is diagonal, with entries $n/q$ and $n/(2q^2)$. At $n=5$ and $\hat{q}=2$, $$ I_5(\hat{\mu},\hat{q})= \begin{pmatrix} 5/2 & 0\\ 0 & 5/(2\cdot 2^2) \end{pmatrix} = \begin{pmatrix} 2.5 & 0\\ 0 & 0.625 \end{pmatrix}. $$

Invert and extract standard errors. The inverse is $$ I_5(\hat{\mu},\hat{q})^{-1}= \begin{pmatrix} 1/2.5 & 0\\ 0 & 1/0.625 \end{pmatrix} = \begin{pmatrix} 0.4 & 0\\ 0 & 1.6 \end{pmatrix}. $$ Thus $\operatorname{SE}(\hat{\mu})\approx\sqrt{0.4}=0.6325$, and $\operatorname{SE}(\hat{q})\approx\sqrt{1.6}=1.2649$, rounded to four decimal places.

Check and trap avoided. The observed information at the MLE gives the same matrix in this model. The variance estimate is $2$, but its standard error is not the standard error of $\hat{\sigma}$; the parameter estimated here is $q=\sigma^2$. Also, using the unbiased sample variance with divisor $n-1$ would not be the MLE used in this information calculation.

Worked Example: Inverting Logistic-Model Information

A hypothetical insurer models whether a policy's claim severity exceeds a stated threshold. Let $Y=1$ indicate an exceedance and let $x$ be a binary rating indicator. Assume independent Bernoulli outcomes with $\operatorname{logit}(P(Y=1\mid x))=\beta_0+\beta_1x$, where $\operatorname{logit}(p)=\log(p/(1-p))$. Among 20 policies with $x=0$, 10 exceed; among 20 with $x=1$, 15 exceed. Estimate the standard errors of both fitted coefficients.

Fit the coefficients. In the $x=0$ group, $\hat{p}_0=10/20=0.5$, so the fitted log-odds are $\log(0.5/0.5)=0$. Hence $\hat{\beta}_0=0$. In the $x=1$ group, $\hat{p}_1=15/20=0.75$, whose log-odds are $\log(0.75/0.25)=\log 3$. Therefore $\hat{\beta}_1=\log 3\approx1.0986$.

Form the sample information. For independent Bernoulli logistic observations with predictor vector $z=(1,x)^\mathsf{T}$, the information contribution is $p(1-p)zz^\mathsf{T}$. For the $x=0$ group, the total weight is $20(0.5)(0.5)=5$; for the $x=1$ group it is $20(0.75)(0.25)=3.75$. Consequently, $$ J_{40}(\hat{\boldsymbol{\beta}})=I_{40}(\hat{\boldsymbol{\beta}}) =5\begin{pmatrix}1&0\\0&0\end{pmatrix} +3.75\begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}8.75&3.75\\3.75&3.75\end{pmatrix}. $$

Invert the matrix. Its determinant is $8.75(3.75)-(3.75)^2=32.8125-14.0625=18.75$. Therefore, $$ J_{40}(\hat{\boldsymbol{\beta}})^{-1} =\frac{1}{18.75} \begin{pmatrix}3.75&-3.75\\-3.75&8.75\end{pmatrix} = \begin{pmatrix}0.2&-0.2\\-0.2&0.4667\end{pmatrix}, $$ where the displayed matrix is rounded to four decimal places.

Extract and check. The coefficient standard errors are $\sqrt{0.2}=0.4472$ for $\hat{\beta}_0$ and $\sqrt{8.75/18.75}=0.6831$ for $\hat{\beta}_1$, rounded to four decimal places. The negative off-diagonal entry is the estimated covariance between the coefficients; it is not a variance and must not be square-rooted. The positive determinant confirms that the fitted information matrix is invertible.

Common mistake avoided. Taking reciprocals of the two diagonal information entries would produce $1/8.75$ and $1/3.75$, which are not the fitted variances. Because the off-diagonal entries are nonzero, the full matrix must be inverted.

Choosing Expected or Observed Information

Expected information averages curvature over possible samples under the fitted model. Observed information uses the curvature for the sample actually obtained. Both are standard ways to estimate MLE covariance under regularity conditions, and in large samples they are often close. In some models or samples they differ, so a question that specifies one should be answered using that quantity. The calculations above happen to give the same expected and observed information at the MLE; this agreement is not guaranteed in every model.

When the sample consists of independent observations with common distribution, per-observation information can be multiplied by $n$ to obtain total information. If a formula already gives total information, do not multiply again. With non-identically distributed independent observations, add their information contributions, using each observation's model and covariates. In all cases, use the information for the likelihood actually fitted, including its parameterization and sampling assumptions.

Common Errors and Interpretation

Forgetting the sample size. Inverting per-observation information without multiplying by the number of independent observations overstates the variance. Conversely, multiplying an already-total information matrix by $n$ understates it.
Confusing variance and standard error. The inverse information estimates a variance (or covariance matrix). Take a square root only after identifying the relevant variance entry.
Inverting a multivariate matrix incorrectly. The inverse matrix, not the reciprocals of its diagonal entries, gives the covariance estimates when information has off-diagonal terms.
Changing parameterization silently. An information calculation for variance $q=\sigma^2$ gives uncertainty for $q$, not directly for $\sigma$. A transformation requires a separate uncertainty calculation.
Treating an approximation as exact. Information-based standard errors are asymptotic. They can be unreliable with small samples, boundary estimates, model misspecification, or nearly singular information.

Key takeaway: Evaluate total expected or observed information at the fitted parameters, invert it, and take square roots of the relevant diagonal entries. For multiple parameters, preserve the full matrix during inversion; its off-diagonal terms influence every component's estimated standard error.

Check Your Understanding

For each problem, use the stated model and the information for the full sample. Report standard errors to four decimal places where needed.

Sixteen independent hypothetical severities follow an exponential distribution with rate $\lambda$. Their total is 32. Find the MLE of $\lambda$ and its information-based standard error. (Answer: $\hat{\lambda}=0.5$, estimated standard error $=0.1250$.)
Ten independent observations follow $N(\mu,q)$, with both parameters unknown, and the fitted variance MLE is $\hat{q}=9$. Find the information-based standard errors of $\hat{\mu}$ and $\hat{q}$. (Answer: $\operatorname{SE}(\hat{\mu})\approx0.9487$, $\operatorname{SE}(\hat{q})\approx4.0249$.)
In a binary logistic model with intercept and binary indicator, 20 observations at $x=0$ have 10 successes and 20 at $x=1$ have 15 successes. Give the fitted coefficients and their information-based standard errors. (Answer: $\hat{\beta}_0=0$, $\hat{\beta}_1=\log 3\approx1.0986$; standard errors approximately $0.4472$ and $0.6831$, respectively.)
A two-parameter model has fitted total information matrix $\begin{pmatrix}6&2\\2&4\end{pmatrix}$. Invert it and give the two standard errors and the estimated covariance. (Answer: inverse $\begin{pmatrix}0.2&-0.1\\-0.1&0.3\end{pmatrix}$; standard errors $\sqrt{0.2}\approx0.4472$ and $\sqrt{0.3}\approx0.5477$; covariance $-0.1$.)

What You'll Learn