Why Information Adds Across Independent Observations
When observations are independent, their likelihoods multiply. Taking logarithms turns that product into a sum, and the sample score becomes a sum of individual scores. Fisher information, however, is an expected squared score, so additivity also depends on the cross-products between different observations having expectation zero. Independence and a regularity condition that makes each score have mean zero provide that justification.
The result is useful whenever a sample contains repeated observations from one model or independent observations from different models. It tells us how to combine the information each observation contributes about the same unknown parameter. It is aligned with the SOA VEE Mathematical Statistics syllabus.
The subscript in \(I_1\) emphasizes that this is information from one observation; it does not mean the parameter equals one. For a sample, we will use \(I_n(\theta)\) for the information in all \(n\) observations. These are expected quantities under the model at the stated value of \(\theta\). They are not generally the same as a score or curvature calculated from one particular realized data set.
The formulas in this tutorial assume that the observations are independent, the parameter is scalar, and each observation has a valid differentiable model. For the zero-mean-score argument, the support must not change with \(\theta\), and differentiation must be allowed to pass through the sum or integral defining total probability. The information terms must be finite. If these assumptions fail, additivity cannot be justified by the argument below without further analysis.
Deriving the Additivity Result
Let \(X_1,\ldots,X_n\) be independent observations with probability mass functions or densities \(f_i(x_i;\theta)\). The observations need not have the same distribution. Independence gives the joint likelihood as a product, and the log-likelihood as a sum:
Define the individual score \(U_i(\theta)=\frac{\partial}{\partial\theta}\log f_i(X_i;\theta)\). Differentiating the sum shows that the joint score is \(\sum_{i=1}^n U_i(\theta)\). The justification for additivity follows in three steps.
For each observation, normalization gives \(\sum_x f_i(x;\theta)=1\) in the discrete case or \(\int f_i(x;\theta)\,dx=1\) in the continuous case. With fixed support and permitted interchange of differentiation and summation or integration, differentiating this identity gives \(E_\theta[U_i(\theta)]=0\).
The square of the sum contains individual squares and cross-products: \(\left(\sum_i U_i\right)^2=\sum_i U_i^2+2\sum_{i<j}U_iU_j\).
For distinct \(i\) and \(j\), independence gives \(E_\theta[U_iU_j]=E_\theta[U_i]E_\theta[U_j]=0\). Taking expectations of the expansion therefore leaves only the individual information terms.
Thus the information in the joint sample is the sum of the information in its observations:
This also explains the expected negative-second-derivative version. The joint log-likelihood is a sum, so its second derivative is a sum of individual second derivatives. Taking the negative expectation yields the sum of individual negative expected second derivatives. When the regularity conditions make that definition equivalent to expected squared score for each observation, it is also equivalent for the sample.
Identical distribution is not required for the general sum. It is required only for the shortcut of multiplying one common information value by \(n\). When observation models differ, calculate each contribution under its own model and add them.
Worked Applications
Worked Example: Information in a Bernoulli Sample
A hypothetical monitoring study records whether each of 8 independent devices triggers an alert. Model each outcome as Bernoulli with common unknown alert probability \(p\), where \(0<p<1\). Calculate the Fisher information in the sample at \(p=0.4\). The realized outcomes are 1, 1, 0, 1, 0, 0, 1, 0.
Method. Find the score and information for one Bernoulli observation, then apply additivity. The Bernoulli mass function is \(f(x;p)=p^x(1-p)^{1-x}\) for \(x\in\{0,1\}\).
The log-likelihood for one observation and its score are
Since \(E_p[X]=p\) and \(\operatorname{Var}_p(X)=p(1-p)\), the score has mean zero and
At \(p=0.4\), one observation contributes
There are 8 independent observations with the same model, so
The result is \(33.3333\), to 4 decimal places. The listed outcomes do not change the Fisher information at the specified parameter value: it is an expectation under the model, not the score calculated from the realized alerts. The outcomes sum to 4, which is consistent with the calculation but is not needed for expected information. Common mistake avoided: treating the observed number of alerts as the sample size; the sample size is 8, whether an observation is 0 or 1.
Worked Example: Combining Normal Measurements With Different Variances
A hypothetical instrument estimates an unknown calibration level \(\theta\). Four independent readings have distribution \(N(\theta,4)\), and three other independent readings have distribution \(N(\theta,9)\). The second argument is the variance. Calculate the total Fisher information about \(\theta\).
Method. For \(X\sim N(\theta,\sigma^2)\) with known variance \(\sigma^2\), the log-density, apart from constants not depending on \(\theta\), is \(-\frac{(x-\theta)^2}{2\sigma^2}\). Its score is
Because \(E_\theta[(X-\theta)^2]=\sigma^2\), the information in one reading is
Each of the four readings with variance 4 contributes \(1/4\). Their combined contribution is \(4(1/4)=1\). Each of the three readings with variance 9 contributes \(1/9\). Their combined contribution is \(3(1/9)=1/3\). By additivity,
The total information is \(1.3333\), to 4 decimal places. The noisier readings contribute less information per observation because their variance is larger. Common mistake avoided: multiplying seven observations by one per-observation information value. The variances differ, so the contributions must be calculated separately before adding.
Worked Example: Poisson Counts With Different Exposure Times
A hypothetical claims-monitoring process records counts \(X_1,X_2,X_3\) over intervals with exposure times \(t_1=1\), \(t_2=2\), and \(t_3=3\) days. Assume independence and \(X_i\sim\operatorname{Poisson}(t_i\lambda)\), where the unknown event rate is \(\lambda>0\) per day. The observed counts are 0, 1, and 3. Calculate the Fisher information at \(\lambda=0.5\) events per day.
Method. For one observation with exposure \(t\), the probability mass function is \(f(x;\lambda)=e^{-t\lambda}(t\lambda)^x/x!\). Its log-likelihood and score are
For a Poisson variable with mean \(t\lambda\), \(E_\lambda[X]=t\lambda\) and \(\operatorname{Var}_\lambda(X)=t\lambda\). Therefore \(E_\lambda[U]= -t+(t\lambda)/\lambda=0\), and
The three exposures contribute \(1/0.5=2\), \(2/0.5=4\), and \(3/0.5=6\), respectively. Thus
The information is \(12\), to 4 decimal places. As a check distinguishing observed score from expected information, the realized total count is \(0+1+3=4\), and the joint score at \(\lambda=0.5\) is \(4/0.5-(1+2+3)=8-6=2\). That score is not the Fisher information. Common mistake avoided: treating each count as \(\operatorname{Poisson}(\lambda)\) and ignoring exposure. Information depends on the interval lengths through their sum, which is 6 days.
Using the Result Correctly
The calculation can be organized as a short sequence: identify the parameter and each observation model, calculate the information contributed by one observation of each type, and add the contributions. For an identically distributed sample, the individual contribution can be multiplied by the sample size. For independent observations with different models, exposures, or known variances, use the sum of their separate contributions instead.
| Sample structure | Total information | What to check |
|---|---|---|
| Independent, identically distributed observations | \(nI_1(\theta)\) | Every observation has the same model and parameter. |
| Independent observations with different models | \(\sum_{i=1}^n I_i(\theta)\) | Calculate each contribution under its own model. |
| Dependent observations | Not justified by this additivity result | Cross-score terms need not have expectation zero. |
A frequent exam trap is to assume that “a sample of size \(n\)” automatically means total information is \(n\) times a single value. That shortcut is valid only when the observations are identically distributed and each has the same parameterization. Independence alone supports the sum, not the common-value shortcut. Another trap is to confuse the expected information with the realized score: the score depends on the observed values, while Fisher information is an expectation under the assumed model.
Additivity also gives an intuitive check on calculations: adding another independent observation with positive information increases total information. If a proposed calculation decreases information after adding an observation, revisit the model and the individual contributions. This observation does not replace checking the assumptions; it is a useful diagnostic once the independent-sample model is appropriate.
Check Your Understanding
For each problem, assume the observations are independent and the stated models hold.
- A hypothetical quality check records 10 independent Bernoulli outcomes with common success probability \(p=0.2\). Find the total Fisher information about \(p\). (Answer: \(62.5\))
- Five independent readings have distribution \(N(\theta,2.5)\), with known variance 2.5 and unknown mean \(\theta\). Find their total Fisher information about \(\theta\). (Answer: \(2\))
- Six independent observations follow an exponential distribution with unknown rate \(\lambda=2\), using density \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x>0\). Find the total Fisher information about \(\lambda\). (Answer: \(1.5\))
- Independent Poisson counts are observed over exposure times 0.5, 1.5, and 4 days, with \(X_i\sim\operatorname{Poisson}(t_i\lambda)\) and \(\lambda=2\) per day. Find the total Fisher information about \(\lambda\). (Answer: \(3\))
- At \(\theta=2\), a sample consists of two independent exponential observations with rate \(\theta\), and three independent normal observations with distribution \(N(\theta,1)\). Find the total Fisher information about \(\theta\). (Answer: \(3.5\))