Why Two Information Measures?
A likelihood describes how compatible a parameter value is with the data. Its curvature near a candidate value indicates how sharply the likelihood distinguishes nearby parameter values: greater curvature means the likelihood is more concentrated locally. There are two useful ways to quantify that curvature. Observed information uses the curvature from the sample actually collected; expected information averages curvature over samples generated by the model.
These measures answer related but different questions. Expected information describes the average curvature the model would produce at a specified parameter value. Observed information describes the curvature in one realized data set. They need not be equal in a finite sample. In particular, evaluating both at a maximum likelihood estimate does not generally make them equal.
Both definitions concern curvature with respect to the same parameterization. Observed information varies with the data; expected information is a model quantity at a specified parameter value. In practice, it is common to compare the observed information at the estimate, \(j(\hat\theta;X)\), with the expected information evaluated at that same estimate, \(I(\hat\theta)\). The latter is a plug-in estimate of expected information, not the expectation taken under an unknown true parameter.
When the log-likelihood is locally concave, positive information corresponds to downward curvature of the log-likelihood. Its reciprocal, \(1/j(\hat\theta;X)\) or \(1/I(\hat\theta)\), is often used as a local curvature-based indication of parameter uncertainty. This comparison is useful for diagnosing sensitivity to the realized sample. It does not, by itself, establish a finite-sample variance or a standard error.
Expected Information as an Average of Curvature
The expected information can be calculated by taking the expectation of the observed information under the model. The order matters: first form the second derivative for a general data set, then average it using the model probabilities at the parameter value of interest. The following categorical model makes the distinction concrete.
Suppose each independent observation has one of three outcomes, with probabilities \(p_1(\theta)=(1+\theta)/4\), \(p_2(\theta)=(1-\theta)/4\), and \(p_3(\theta)=1/2\), where \(-1<\theta<1\). For a sample of size \(n\), let \(N_k\) be the number of observations in outcome \(k\). The counts satisfy \(N_1+N_2+N_3=n\). Ignoring factors that do not involve \(\theta\), the log-likelihood is
Differentiating twice gives the observed information. Averaging that expression requires using \(E_\theta[N_1]=np_1(\theta)\) and \(E_\theta[N_2]=np_2(\theta)\), since each count is the sum of \(n\) indicator variables with the relevant outcome probability.
The first derivative is \(N_1/(1+\theta)-N_2/(1-\theta)\). The second derivative is \(-N_1/(1+\theta)^2-N_2/(1-\theta)^2\), so the observed information is \(N_1/(1+\theta)^2+N_2/(1-\theta)^2\).
Substitute \(E_\theta[N_1]=n(1+\theta)/4\) and \(E_\theta[N_2]=n(1-\theta)/4\). This gives \(I(\theta)=n/[4(1+\theta)]+n/[4(1-\theta)]\).
Combining the two fractions yields \(I(\theta)=n/[2(1-\theta^2)]\). This is the average observed curvature under the model, not the curvature that every sample must produce.
This calculation explains why the two measures can differ: the observed information depends on the realized counts, whereas expected information replaces those counts by their model averages. The derivation assumes independent observations from the stated model and an interior parameter value. It does not apply at the excluded endpoints \(\theta=-1\) or \(\theta=1\), where the model probabilities reach the boundary of the parameter space.
Worked Comparisons
Worked Example: Poisson Information at a Fixed Parameter
A hypothetical review team records five independent event counts with values \(0,1,1,2,4\). Assume each count has a Poisson distribution with mean \(\lambda\). Compare observed and expected information at \(\lambda=2\), then compare them again at the maximum likelihood estimate.
Method. For independent Poisson observations \(X_1,\ldots,X_n\), the log-likelihood, apart from terms not involving \(\lambda\), is \(\ell(\lambda)=\sum_i X_i\log\lambda-n\lambda\). Its second derivative is \(-\sum_iX_i/\lambda^2\). Therefore \(j(\lambda;X)=\sum_iX_i/\lambda^2\). Since \(E_\lambda[\sum_iX_i]=n\lambda\), the expected information is \(I(\lambda)=n/\lambda\).
At \(\lambda=2\). There are \(n=5\) observations and their sum is \(0+1+1+2+4=8\). Thus \(j(2;X)=8/2^2=8/4=2\), while \(I(2)=5/2=2.5\). The observed sample has less curvature than the model's expected curvature at this fixed value.
At the maximum likelihood estimate. The estimate is the sample mean, so \(\hat\lambda=8/5=1.6\). Then \(j(1.6;X)=8/(1.6)^2=8/2.56=3.125\), and \(I(1.6)=5/1.6=3.125\). They agree here because \(\sum_iX_i=n\hat\lambda\), so both expressions reduce to \(n/\hat\lambda\).
Check. At the fixed value, the two calculations used different quantities: the observed total of 8 versus its model expectation \(5(2)=10\). At the estimate, the fitted mean matches the observed sample mean, explaining the equality. The common mistake is to infer from this Poisson example that observed and expected information always agree at the estimate; the next example shows that they need not.
Worked Example: A Finite-Sample Difference in a Three-Outcome Model
A hypothetical monitoring process records ten independent outcomes: outcome 1 occurs three times, outcome 2 once, and outcome 3 six times. Use the three-outcome model defined above. Find the maximum likelihood estimate, the observed information at that estimate, and the expected information evaluated at that estimate.
Step 1: Estimate the parameter. Write \(m=N_1+N_2=3+1=4\). The score is \(N_1/(1+\theta)-N_2/(1-\theta)\). Setting it equal to zero gives \(N_1(1-\theta)=N_2(1+\theta)\), hence \(\hat\theta=(N_1-N_2)/(N_1+N_2)=(3-1)/4=0.5\). Both informative outcome counts are positive, and this estimate is inside the allowed interval.
Step 2: Calculate observed information. At \(\hat\theta=0.5\), \(j(\hat\theta;X)=3/(1.5)^2+1/(0.5)^2=3/2.25+1/0.25=4/3+4=16/3\), or approximately \(5.3333\).
Step 3: Calculate expected information at the estimate. Here \(n=10\), so \(I(\hat\theta)=10/[2(1-(0.5)^2)]=10/[2(0.75)]=10/1.5=20/3\), or approximately \(6.6667\).
Interpretation and check. The plug-in expected information is \(20/3\), larger than the observed \(16/3\). Their reciprocals are \(3/20=0.15\) and \(3/16=0.1875\), respectively, so the observed curvature suggests greater local uncertainty by this measure. The ratio \( (16/3)/(20/3)=0.8\) also agrees with \(2m/n=8/10=0.8\). The common mistake is to replace \(n\) by the number of outcomes with parameter-dependent probabilities in the expected-information formula; the expectation is over all ten observations, including outcome 3.
Worked Example: How the Realized Counts Change the Comparison
Consider another hypothetical sample of ten outcomes from the same model: outcome 1 occurs five times, outcome 2 occurs three times, and outcome 3 occurs twice. Calculate both information measures at the maximum likelihood estimate and state which gives the larger reciprocal curvature measure.
Step 1: Find the estimate. Here \(m=5+3=8\), so \(\hat\theta=(5-3)/8=0.25\).
Step 2: Find observed information. Substituting into the observed-information formula gives \(j(\hat\theta;X)=5/(1.25)^2+3/(0.75)^2=5/1.5625+3/0.5625=3.2+16/3=128/15\), approximately \(8.5333\).
Step 3: Find expected information at the estimate. \(I(\hat\theta)=10/[2(1-(0.25)^2)]=10/[2(0.9375)]=10/1.875=16/3\), approximately \(5.3333\).
Step 4: Compare reciprocals. The observed-information reciprocal is \(15/128\), approximately \(0.1172\); the plug-in expected-information reciprocal is \(3/16=0.1875\). Thus observed information indicates greater local curvature, and the smaller reciprocal, in this sample. The check is that \( (128/15)/(16/3)=1.6=2m/n=16/10\). Unlike the prior sample, a larger share of observations fell in the two parameter-dependent categories, increasing realized curvature relative to its plug-in expectation.
Interpreting the Difference Carefully
Observed and expected information are both measures of likelihood curvature, but they are not interchangeable descriptions of a particular sample. Expected information averages across possible samples under the assumed model. Observed information reports what the realized data contribute at a chosen parameter value. The difference may be especially visible in small samples, where realized counts can differ substantially from their model averages.
The Poisson example illustrates an important special feature: observed and expected information coincide at the maximum likelihood estimate for that model. In the three-outcome model, the estimate solves the score equation, but the observed and expected information at that estimate can still differ. A zero score fixes the local slope; it does not force the observed curvature to equal the model's average curvature.
| Quantity | Depends on | Interpretation |
|---|---|---|
| Observed information \(j(\theta;X)\) | The realized data and the evaluation value \(\theta\) | Curvature in this sample |
| Expected information \(I(\theta)\) | The model and the evaluation value \(\theta\) | Average curvature across samples generated at \(\theta\) |
| Plug-in expected information \(I(\hat\theta)\) | The model and the fitted value | Expected curvature estimated using the fitted parameter |
For a regular model with a well-defined, interior maximum, both measures are typically evaluated at \(\hat\theta\) when comparing curvature-based assessments. Their reciprocals can be useful indicators of local parameter precision, but they are not automatically exact finite-sample variances. This tutorial's comparison is about the information calculations themselves; turning information into an estimated standard error requires additional inferential steps.
Common Exam Traps
- Confusing the data-specific and model-average quantities. Observed information uses the actual data in the second derivative. Expected information takes an expectation under the model.
- Mixing evaluation points. A comparison of \(j(\hat\theta;X)\) with \(I(\theta_0)\) combines different parameter values. State clearly whether both are evaluated at a fixed value or both at the estimate.
- Assuming equality at the maximum likelihood estimate. The score being zero is a first-derivative condition. It does not generally make observed and expected second derivatives equal.
- Forgetting what is random. For a fixed parameter, expected information is not a random function of the observed counts; observed information is.
- Overinterpreting the reciprocal. A reciprocal-information quantity is a local curvature-based uncertainty measure, not necessarily the exact finite-sample variance of the estimator.
Check Your Understanding
For each problem, distinguish the realized curvature from expected curvature and keep the stated evaluation point fixed.
- Six independent Poisson counts have total 9. At \(\lambda=3\), calculate observed and expected information. Then give both values at the maximum likelihood estimate. (Answer: At \(3\), \(j=9/9=1\) and \(I=6/3=2\); at \(\hat\lambda=1.5\), both equal \(6/1.5=4\).)
- In the three-outcome model, a sample has counts \((N_1,N_2,N_3)=(2,2,4)\). Find \(\hat\theta\), observed information at the estimate, and expected information at the estimate. (Answer: \(\hat\theta=0\), \(j=4\), and \(I=8/2=4\).)
- In the three-outcome model, counts are \((4,1,2)\). Find the maximum likelihood estimate and compare observed information with expected information at that estimate. (Answer: \(\hat\theta=3/5=0.6\), \(j=125/16=7.8125\), and \(I=175/32=5.46875\); observed information is larger.)
- For the three-outcome model with \(n=12\), evaluate expected information at \(\theta=0.2\). A realized sample has counts \((3,2,7)\); calculate observed information at \(\theta=0.2\) and compare. (Answer: \(I(0.2)=6.25\), while \(j(0.2;X)=3/1.44+2/0.64=125/24\approx5.2083\); expected information is larger.)
- In the three-outcome model, counts are \((6,2,4)\). Calculate \(\hat\theta\), both information measures at that estimate, and their reciprocals. (Answer: \(\hat\theta=0.5\), \(j=32/3\), \(I=8\), \(1/j=3/32\), and \(1/I=1/8\).)