Additivity of Information Across Independent Data

Why Information Adds Across Independent Data

When independent observations inform the same parameter, their likelihood contributions multiply. Taking logarithms turns that product into a sum, so the score—the derivative of the log-likelihood—is also a sum. The key question is whether the expected Fisher information is the sum of the separate expected contributions. Under regularity conditions, it is: the cross-products between different observations’ scores have expectation zero.

This result applies even when observations are not identically distributed. An observation with a different variance, success probability, or known exposure can contribute a different amount of information. Additivity says to sum those contributions; it does not say that every observation contributes equally.

Definition: For a scalar parameter \(\theta\), let \(X_i\) have density or mass function \(f_i(x_i;\theta)\) with respect to a fixed dominating measure. The individual score is \(U_i(\theta)=\partial \log f_i(X_i;\theta)/\partial\theta\). When the score has a finite second moment and \(E_\theta[U_i(\theta)]=0\), the expected Fisher information in \(X_i\) is \(I_i(\theta)=E_\theta[U_i(\theta)^2]\). For independent observations \(X_1,\ldots,X_n\), write \(L(\theta)\) for their joint likelihood and \(\ell(\theta)=\log L(\theta)\). Their expected information is \(I(\theta)=E_\theta[(\partial\ell(\theta)/\partial\theta)^2]\).

Throughout the scalar result below, assume \(\theta\) lies in an open interval and the observations are independent. For each observation, assume the model is differentiable in \(\theta\), the density or mass function is normalized, and differentiation can be interchanged with integration or summation. In particular, the regularity conditions must imply \(E_\theta[U_i(\theta)]=0\). Assume also that the scores have finite second moments. These conditions rule out, for example, some parameter-dependent support and boundary cases where the usual score identities need not hold.

Additivity for Independent Observations

Independence makes the joint likelihood a product, whether or not the distributions are identical. The result below gives the precise additivity statement and its justification.

Theorem: Under the regularity conditions above, if \(X_1,\ldots,X_n\) are independent and all their distributions depend on the same scalar parameter \(\theta\), then \(I(\theta)=\sum_{i=1}^n I_i(\theta)\).

Factor the likelihood.
Independence gives \(L(\theta)=\prod_{i=1}^n f_i(X_i;\theta)\), so \(\ell(\theta)=\sum_{i=1}^n\log f_i(X_i;\theta)\).

Add the scores.
Differentiating the finite sum gives the joint score \(U(\theta)=\partial\ell(\theta)/\partial\theta=\sum_{i=1}^n U_i(\theta)\).

Expand its second moment.
\(I(\theta)=E_\theta[U(\theta)^2]=\sum_i E_\theta[U_i(\theta)^2]+2\sum_{i<j}E_\theta[U_i(\theta)U_j(\theta)]\).

Remove the cross-terms.
For \(i\ne j\), independence and finite score moments imply \(E_\theta[U_iU_j]=E_\theta[U_i]E_\theta[U_j]=0\). Thus only the individual second moments remain.

For completeness, the zero-mean score follows from normalization and the stated differentiation condition. If \(\mu_i\) is the dominating measure, then \(\int f_i(x;\theta)\,d\mu_i(x)=1\). Differentiating both sides with respect to \(\theta\) gives \(\int \partial f_i(x;\theta)/\partial\theta\,d\mu_i(x)=0\). Wherever the score is defined, \(\partial f_i/\partial\theta=f_i\,\partial\log f_i/\partial\theta\), so the integral is \(E_\theta[U_i(\theta)]=0\). The cross-term calculation in Step 4 then applies, and the remaining terms are exactly \(I_i(\theta)\). Therefore \(I(\theta)=\sum_i I_i(\theta)\). \(\square\)

The conclusion does not require identical distributions. It requires a common parameter and independent observations whose individual information contributions are defined under suitable regularity conditions. If observations are dependent, the expansion still contains cross-terms, but independence no longer makes them vanish automatically.

Vector Parameters

The same reasoning applies when the common parameter is a vector. This form is useful when a model has several parameters, but the additivity concerns only information from independent observations; it does not change how parameters themselves are represented.

Theorem: Let \(\theta\) be a vector in an open subset of \(\mathbb{R}^p\), and let \(U_i(\theta)\) be the \(p\)-component score vector for observation \(i\). If the observations are independent, each score has mean zero, and each score has finite second moments, then the expected information matrix of the joint data is \( \mathcal{I}(\theta)=\sum_{i=1}^n\mathcal{I}_i(\theta)\), where \(\mathcal{I}_i(\theta)=E_\theta[U_i(\theta)U_i(\theta)^\mathsf{T}]\).

Indeed, the joint score vector is \(U=\sum_i U_i\). Expanding its outer product yields \(E_\theta[UU^\mathsf{T}]=\sum_i E_\theta[U_iU_i^\mathsf{T}]+\sum_{i\ne j}E_\theta[U_iU_j^\mathsf{T}]\). For \(i\ne j\), independence gives \(E_\theta[U_iU_j^\mathsf{T}]=E_\theta[U_i]E_\theta[U_j]^\mathsf{T}=0\). The remaining terms are the individual information matrices, proving the result. \(\square\)

Worked Examples

Worked Example: Bernoulli Groups With Different Sample Sizes

Suppose independent groups provide \(m_1=10\) and \(m_2=15\) Bernoulli observations, all with the same unknown success probability \(p\), where \(0<p<1\). Calculate the total expected information about \(p\).

For one Bernoulli observation \(X\), the mass function is \(f(x;p)=p^x(1-p)^{1-x}\), for \(x\in\{0,1\}\). Its log-likelihood and score are \(\ell(p)=x\log p+(1-x)\log(1-p)\) and \(U(p)=x/p-(1-x)/(1-p)=(x-p)/(p(1-p))\). Since \(E_p[X]=p\) and \(\operatorname{Var}_p(X)=p(1-p)\), \(E_p[U(p)^2]=E_p[(X-p)^2]/[p^2(1-p)^2]=1/[p(1-p)]\).

Each group consists of independent observations, so the first group contributes \(10/[p(1-p)]\) and the second contributes \(15/[p(1-p)]\). Additivity gives \(I(p)=25/[p(1-p)]\). At \(p=0.4\), this is \(25/(0.4\cdot0.6)=25/0.24\approx104.17\). The calculation uses the total of 25 Bernoulli observations, not the number of groups.

Worked Example: Poisson Counts With Known Exposures

Suppose \(Y_1,Y_2,Y_3\) are independent, with \(Y_i\sim\operatorname{Poisson}(t_i\lambda)\), where the known exposures are \(t_1=1\), \(t_2=2\), and \(t_3=4\), and the unknown rate satisfies \(\lambda>0\). Find the total information about \(\lambda\).

For one count with exposure \(t_i\), its log mass is \(\ell_i(\lambda)=Y_i\log(t_i\lambda)-t_i\lambda-\log(Y_i!)\). Differentiation gives \(U_i(\lambda)=Y_i/\lambda-t_i\). Because \(E_\lambda[Y_i]=t_i\lambda\) and \(\operatorname{Var}_\lambda(Y_i)=t_i\lambda\), the score has mean zero and \(I_i(\lambda)=E_\lambda[U_i(\lambda)^2]=\operatorname{Var}_\lambda(Y_i)/\lambda^2=t_i/\lambda\).

Thus the three contributions are \(1/\lambda\), \(2/\lambda\), and \(4/\lambda\), giving \(I(\lambda)=(1+2+4)/\lambda=7/\lambda\). For example, at \(\lambda=2\), the total information is \(7/2=3.5\). The total exposure is 7, so the longer-exposure observations contribute more information than the shorter-exposure observation. The exposures are treated as fixed known quantities in this calculation.

Worked Example: Normal Observations With Different Known Variances

Let \(X_1,X_2,X_3\) be independent normal observations with common unknown mean \(\mu\), and known variances \(4,4,\) and \(1\), respectively. Calculate their total information about \(\mu\).

For \(X_i\sim N(\mu,\sigma_i^2)\), the log density, up to constants that do not depend on \(\mu\), is \(- (X_i-\mu)^2/(2\sigma_i^2)\). Its score is \(U_i(\mu)=(X_i-\mu)/\sigma_i^2\). The score has mean zero, and its second moment is \(I_i(\mu)=E_\mu[(X_i-\mu)^2]/\sigma_i^4=\sigma_i^2/\sigma_i^4=1/\sigma_i^2\).

The first two observations each contribute \(1/4\), and the third contributes \(1\). Therefore \(I(\mu)=1/4+1/4+1=3/2\). The third observation carries more information about the mean because its known variance is smaller. Adding the three observations as though they had equal information would incorrectly give a total of 3 times one common contribution.

What Additivity Does—and Does Not—Say

Additivity is a rule for combining information about the same parameter from independent observations. It is not a claim that information is determined by the number of records alone. In the Bernoulli example, each trial has the same contribution when \(p\) is common; in the Poisson example, contribution is proportional to known exposure; in the normal example, it is inversely proportional to known variance.

A useful way to check a calculation is to compute each observation’s score, verify that its expected score is zero, and then sum its expected squared score. If the data are grouped, count the number of independent observations represented by each group and multiply that group’s per-observation information by its size, provided those observations share the same distribution. For example, a group of \(m\) independent observations each with information \(I_0(\theta)\) contributes \(mI_0(\theta)\).

Mistake: “Independent observations always add information, even if their score means are not zero.” The joint score’s second moment includes cross-products. The correction is to verify the regularity conditions that give mean-zero scores; independence then makes the cross-products vanish.
Mistake: “Every independent observation contributes the same amount.” This is true only when the individual information contributions are equal. Nonidentical distributions can contribute different amounts, as the exposure and variance examples show.
Mistake: “The number of groups is the sample size.” A group may represent many independent observations. Sum contributions over observations, or multiply a per-observation contribution by the group’s actual size when appropriate.

Key takeaway: Under regularity conditions, independent observations with a common parameter have additive expected Fisher information. The total is the sum of their individual contributions, which may differ when the observations have different distributions or known exposures.

Check Your Understanding

For each question, identify the individual information contributions before combining them.

What regularity fact makes the cross-products of independent scores have expectation zero?
Four independent Bernoulli observations share an unknown success probability \(p\). What is their total expected information about \(p\)?
Independent Poisson counts have means \(3\lambda\) and \(5\lambda\), with known exposures. What is their total expected information about \(\lambda\)?
Two independent normal observations share an unknown mean and have known variances \(2\) and \(8\). What is their total information about the mean?
Why does independence by itself not justify dropping score cross-terms if the individual scores do not have mean zero?