Additivity of Fisher Information

Why Information Adds Across Independent Data

When a parameter is estimated from several independent observations, each observation contributes evidence about that parameter. Fisher information provides a numerical measure of that evidence. The key practical result is that, under standard regularity conditions, the expected information in the full sample is the sum of the information contributions from its independent parts. This remains useful when groups have different observation counts or follow different distributions, provided their likelihoods depend on the parameter being studied.

The result applies to expected Fisher information. It is not a claim that every realized data set contributes the same amount of information, nor does it say that one should average the information contributions when combining samples. The total is a sum.

Definition: Let an observation \(X_i\) have density or probability mass function \(f_i(x;\theta)\), where \(\theta\) is a scalar parameter. Its score is \(U_i(\theta)=\frac{\partial}{\partial\theta}\log f_i(X_i;\theta)\). Under the usual regularity conditions, its Fisher information is \(I_i(\theta)=E_\theta[U_i(\theta)^2]\). For a parameter vector \(\boldsymbol{\theta}\), the score is the gradient of the log-likelihood and the information matrix is \(I_i(\boldsymbol{\theta})=E_{\boldsymbol{\theta}}[U_iU_i^{T}]\).

The expectation in these definitions is taken under the model at the parameter value being considered. Fisher information is therefore a function of the parameter, even when a particular calculation substitutes a numerical value. For a scalar parameter, information is nonnegative. For a vector parameter, the information matrix is positive semidefinite.

Additivity: Result and Justification

Suppose \(X_1,\ldots,X_n\) are independent, and their distributions may differ. Write their individual log-likelihood contributions as \(\ell_i(\theta)=\log f_i(X_i;\theta)\). Independence makes the joint likelihood a product, so its log-likelihood is a sum. Differentiating that sum gives a total score that is the sum of the individual scores.

Theorem: If \(X_1,\ldots,X_n\) are independent observations whose likelihood contributions depend on the same parameter \(\theta\), and the standard regularity conditions hold, then \(I_{\mathrm{total}}(\theta)=\sum_{i=1}^{n} I_i(\theta)\). For a parameter vector, the corresponding information matrices also add: \(I_{\mathrm{total}}(\boldsymbol{\theta})=\sum_{i=1}^{n} I_i(\boldsymbol{\theta})\).

Here the standard regularity conditions include differentiability of the likelihood with respect to the parameter, a parameter-independent support, and conditions that permit differentiation under the integral or summation sign. The score should have a finite second moment and mean zero. These conditions matter: when the support depends on the parameter or differentiation and expectation cannot be interchanged, familiar score identities may fail. Do not apply an identity derived under regularity conditions without checking that the model supports it.

\ell_{\mathrm{total}}(\theta)=\sum_{i=1}^{n}\ell_i(\theta), \qquad U_{\mathrm{total}}(\theta)=\sum_{i=1}^{n}U_i(\theta).

A short derivation explains why the information sums. Under the regularity conditions, each individual score has mean zero. Independence then makes the expectation of each product of distinct scores zero. Expanding the square of the total score leaves only the individual squared-score terms:

\begin{aligned} I_{\mathrm{total}}(\theta) &=E_\theta\left[\left(\sum_{i=1}^{n}U_i(\theta)\right)^2\right]\\ &=\sum_{i=1}^{n}E_\theta[U_i(\theta)^2] +2\sum_{i<j}E_\theta[U_i(\theta)U_j(\theta)]\\ &=\sum_{i=1}^{n}I_i(\theta) +2\sum_{i<j}E_\theta[U_i(\theta)]E_\theta[U_j(\theta)]\\ &=\sum_{i=1}^{n}I_i(\theta). \end{aligned}

The factorization in the third line follows from independence; the final equality follows because each score has mean zero. The same reasoning works for a parameter vector using score outer products: the cross terms have expectation zero, leaving the sum of the individual information matrices. Alternatively, expected information defined through the negative expected second derivative also adds because the joint log-likelihood is a sum and expectation is linear, when that definition is valid.

Using Additivity With Unequal Sample Sizes

If observations are identically distributed, each contributes the same amount \(I_1(\theta)\), so \(n\) independent observations provide \(nI_1(\theta)\) information. But identical distributions are not required. If one group has four observations and another has six, and their per-observation information differs, multiply each group’s contribution by its own count and add. For a group of \(n_g\) independent observations with common per-observation information \(I_g(\theta)\), its total contribution is \(n_gI_g(\theta)\).

Worked Example: Combining Exponential Samples

Two independent groups of hypothetical claim amounts follow exponential distributions with the same unknown rate \(\theta\), using density \(f(x;\theta)=\theta e^{-\theta x}\) for \(x\geq 0\). Group A has 4 observations and Group B has 3. Calculate the total Fisher information at \(\theta=0.5\). Use the likelihood-score method, and avoid the mistake of averaging the group information.

Step 1: Find the score for one observation. The log-density is \(\ell(\theta)=\log\theta-\theta x\), so its score is \(U(\theta)=1/\theta-x\).

Step 2: Find information per observation. An exponential random variable with rate \(\theta\) has variance \(1/\theta^2\). Since the score is \(1/\theta-X\), its variance is also \(1/\theta^2\). Its mean is zero, so the expected squared score is \(I_1(\theta)=1/\theta^2\).

Step 3: Evaluate at the specified parameter. At \(\theta=0.5\), \(I_1(0.5)=1/(0.5)^2=1/0.25=4\).

Step 4: Add the contributions. Group A contributes \(4(4)=16\), and Group B contributes \(3(4)=12\). Thus \(I_{\mathrm{total}}(0.5)=16+12=28\).

Check and common mistake. There are 7 independent observations, each contributing 4, so \(7(4)=28\), confirming the result. Averaging the group contributions, such as \((16+12)/2=14\), would report an average per group rather than the information in all observations. The method is to sum, not average.

Independent Samples Need Not Have the Same Distribution

Additivity is particularly useful when independent observations have different known exposures or variances. The observations must still provide information about the same parameter for their contributions to be combined into its total information. Calculate each observation’s contribution under its own model, then sum those contributions. The next examples show how the details of a model determine each contribution.

Worked Example: Poisson Counts With Different Exposures

A hypothetical portfolio supplies two independent sets of event counts. Each count in Group A is Poisson with mean \(\lambda t\), where the known exposure is \(t=0.5\); there are 4 such counts. Each count in Group B is Poisson with mean \(\lambda t\), where \(t=2\); there are 6 such counts. The common unknown rate is \(\lambda\). Calculate total information at \(\lambda=2\). The method is to obtain information per count from its Poisson likelihood, then add by group.

Step 1: Derive the contribution from one count. For \(X\sim\operatorname{Poisson}(\lambda t)\), the log-likelihood, apart from terms not involving \(\lambda\), is \(X\log(\lambda t)-\lambda t\). Its score is \(U(\lambda)=X/\lambda-t\).

Step 2: Evaluate the expected squared score. The score has mean zero because \(E[X]=\lambda t\). Its variance, and therefore its expected square, is \(\operatorname{Var}(X)/\lambda^2=(\lambda t)/\lambda^2=t/\lambda\). Thus one count contributes \(t/\lambda\).

Step 3: Calculate each group’s contribution. At \(\lambda=2\), a Group A count contributes \(0.5/2=0.25\), so the 4 counts contribute \(4(0.25)=1\). A Group B count contributes \(2/2=1\), so the 6 counts contribute \(6(1)=6\).

Step 4: Add and check. The total is \(1+6=7\). Equivalently, total exposure is \(4(0.5)+6(2)=2+12=14\), and the total information is \(14/2=7\). This check confirms that the counts and exposures have both been included. A common mistake is to treat all 10 counts as if they had the same exposure.

Worked Example: Normal Observations With Different Known Variances

A hypothetical calibration study observes a common unknown mean \(\mu\). Four independent readings have distribution \(N(\mu,4)\), and three independent readings have distribution \(N(\mu,1)\); the second argument is the variance, and both variances are known. Calculate the total information about \(\mu\), evaluating at \(\mu=10\). The method is to find the information for a normal mean in each variance group and add the group totals.

Step 1: Find the contribution for one reading. If \(X\sim N(\mu,\sigma^2)\) with known \(\sigma^2\), the score for \(\mu\) is \(U(\mu)=(X-\mu)/\sigma^2\). Its expected square is \(E[(X-\mu)^2]/\sigma^4=\sigma^2/\sigma^4=1/\sigma^2\). The contribution does not depend on the value of \(\mu\), so evaluating at 10 does not change it.

Step 2: Calculate each group’s information. In the first group, each variance is 4, so each reading contributes \(1/4\); four readings contribute \(4(1/4)=1\). In the second group, each variance is 1, so each reading contributes \(1\); three readings contribute \(3(1)=3\).

Step 3: Add and verify. Total information is \(1+3=4\). As a direct check, the individual contributions are four copies of \(1/4\) and three copies of 1, whose sum is \(4/4+3=4\). A common mistake is to use one average variance for every reading; that discards the fact that the two groups have different precision.

Common Exam Trap: Total Versus Average Information

The distinction between total and average information is easy to lose when a problem gives information per observation or information per group. If the question asks for the information in the full sample, multiply per-observation contributions by their observation counts and sum. If it asks for the average information per observation, divide the total by the total number of observations only after summing.

\text{Total information}=\sum_g n_g I_g(\theta), \qquad \text{average information per observation} =\frac{\sum_g n_g I_g(\theta)}{\sum_g n_g}.

For the exponential example, the total is 28, whereas the average per observation is \(28/7=4\). Neither number is intrinsically wrong; they answer different questions. Reporting 4 as the total would understate the full-sample information by a factor of 7.

Also check whether observations are actually independent. If they are dependent, the cross terms in the squared total score need not be zero, and simply summing individual information contributions may be incorrect. Likewise, if separate data sets concern different parameters, do not combine their scalar information as though they estimated one common parameter; formulate the relevant parameter vector and information matrix.

Key takeaway: Under the standard regularity conditions, independent observations contribute additive expected Fisher information. Compute each contribution for the model and parameter it actually has, multiply by the correct observation count when appropriate, and sum to obtain information for the full data set.

Check Your Understanding

For each question, identify the information contribution from each independent part before combining contributions.

Five independent observations follow an exponential distribution with unknown rate \(\theta=0.4\). What is the total Fisher information for \(\theta\)? (Answer: 31.25)
Independent Poisson counts share an unknown rate \(\lambda=5\). Three counts each have known exposure 2, and four counts each have known exposure 0.5. Find the total information for \(\lambda\). (Answer: 1.6)
Independent observations share an unknown normal mean \(\mu\). Two observations have known variance 2 and five have known variance 5. Find the total information for \(\mu\). (Answer: 2)
Three independent exponential observations have unknown rate \(\theta=2\), and four independent Poisson counts have unknown rate \(\lambda=8\). The two samples involve distinct parameters. Give the joint Fisher information matrix for \((\theta,\lambda)\). (Answer: \(\begin{pmatrix}0.75&0\\0&0.5\end{pmatrix}\))
Two independent normal groups share an unknown mean. Three observations have known variance 9, and two have known variance 1. Find total information for the mean, and state which group contributes more. (Answer: \(7/3\); the variance-1 group contributes more, with information 2 versus \(1/3\))

What You'll Learn