Adding Independent Information Contributions

Why Independent Information Adds

A study may measure the same model parameters through several independent observation types or data sources. For example, one source may contribute concentration measurements and another a different assay or endpoint. Each source has its own likelihood, but if both are informative about the same parameter vector, their information can be combined. The key condition is not that the measurements have the same units or response distribution; it is that the specified joint likelihood factors into independent source contributions, with each contribution expressed in a common parameterization.

This addition is useful in design calculations because it summarizes how the separate sources jointly constrain the parameters. It does not mean that every reported quantity improves in a simple scalar way: matrices can contain off-diagonal terms describing joint parameter information, and the covariance approximation must be calculated from the combined matrix.

Definition: Let \(\theta\) be a shared parameter vector, and let \(Y_1,\ldots,Y_K\) denote data sources that are independent under the model at \(\theta\). If source \(k\) has log likelihood \(\ell_k(\theta;Y_k)\), score \(U_k(\theta)=\partial \ell_k/\partial\theta\), and Fisher information matrix \(\mathcal I_k(\theta)=E_\theta[U_k U_k^\mathsf{T}]\), then the joint Fisher information is the sum \(\mathcal I_{\mathrm{total}}(\theta)=\sum_{k=1}^K\mathcal I_k(\theta)\), provided the usual regularity conditions hold.

Each source must refer to the same target parameter vector, or its information must first be represented in a clearly defined common parameterization. A source that does not depend on one of the parameters contributes zero information in that parameter's direction. If sources have distinct nuisance parameters, those parameters cannot simply be treated as shared; the joint parameter vector and each source's dependence on it must be specified before adding matrices.

Conditions and Derivation

The basic assumptions are that the source data are independent under the model, the joint likelihood is the product of their likelihoods, and the likelihood is sufficiently regular for the score to have mean zero and for Fisher information to be represented by the score covariance. The parameter support should not introduce irregular boundary effects, and the required derivatives and expectations must exist. These conditions are standard for regular likelihood-based information calculations, but they should be checked rather than inferred merely from the fact that measurements came from different instruments or files.

Independence gives a product likelihood and therefore an additive log likelihood. The total score is a sum of source scores. In the expectation of its outer product, cross-source terms vanish because independent scores with mean zero have zero expected cross-products. This gives the result directly.

L(\theta;Y_1,\ldots,Y_K)=\prod_{k=1}^K L_k(\theta;Y_k), \qquad \ell(\theta)=\sum_{k=1}^K\ell_k(\theta), \qquad U(\theta)=\sum_{k=1}^K U_k(\theta).

For two sources, expand the information defined as the expected outer product of the total score. Since \(Y_1\) and \(Y_2\) are independent and \(E_\theta[U_k]=0\), the cross-products have expectation zero:

\begin{aligned} \mathcal I_{\mathrm{total}}(\theta) &=E_\theta[(U_1+U_2)(U_1+U_2)^\mathsf{T}]\\ &=E_\theta[U_1U_1^\mathsf{T}] +E_\theta[U_2U_2^\mathsf{T}] +E_\theta[U_1U_2^\mathsf{T}] +E_\theta[U_2U_1^\mathsf{T}]\\ &=\mathcal I_1(\theta)+\mathcal I_2(\theta). \end{aligned}

For example, \(E_\theta[U_1U_2^\mathsf{T}]=E_\theta[U_1]E_\theta[U_2]^\mathsf{T}=0\). Repeating the argument for all pairs among \(K\) independent sources proves the stated sum. The same additivity holds for expected information defined as the negative expected Hessian: the joint log likelihood is a sum, so its Hessian is a sum, and expectation preserves addition. The observed information, defined as the negative Hessian of the realized log likelihood, also adds pointwise when that log likelihood factors.

This is a sum of matrices of the same dimension, not a sum of standard errors or variances. The combined information is symmetric and positive semidefinite. If it is positive definite, its inverse is the local covariance approximation commonly used for a regular maximum likelihood estimator: \(\operatorname{Cov}(\hat\theta)\approx\mathcal I_{\mathrm{total}}(\theta)^{-1}\), with the information evaluated at the relevant parameter value. This is an asymptotic or local approximation, not an exact finite-sample identity.

Worked Calculations

Worked Example: Combining Two Scalar Measurement Sources

Suppose two independent hypothetical assays measure the same parameter \(\theta\), a mean concentration in mg/L. Source A provides \(n_A=4\) observations, with hypothetical values 8, 10, 9, and 11 mg/L. Source B provides \(n_B=6\) observations, with hypothetical values 12, 10, 11, 13, 9, and 12 mg/L. Assume the observations in source A are independent \(N(\theta,4)\) and those in source B are independent \(N(\theta,9)\), where the variances are known in \((\mathrm{mg/L})^2\).

For a normal observation with mean \(\theta\) and known variance \(\sigma^2\), the score is \((Y-\theta)/\sigma^2\), so its Fisher information for \(\theta\) is \(1/\sigma^2\). For \(n\) independent observations with the same variance, the information is \(n/\sigma^2\). Thus the two contributions are

\mathcal I_A=\frac{4}{4}=1.000\;(\mathrm{mg/L})^{-2}, \qquad \mathcal I_B=\frac{6}{9}=0.6667\;(\mathrm{mg/L})^{-2}.

The observations are independent by the stated model, share the same mean parameter, and have known variances, so addition is justified. The total information is \(1.000+0.6667=1.6667\;(\mathrm{mg/L})^{-2}\). Its inverse gives the local variance approximation \(1/1.6667=0.6000\;(\mathrm{mg/L})^2\), and the corresponding approximate standard error is \(\sqrt{0.6000}=0.7746\) mg/L. The observed values do not enter this expected-information calculation because the assumed variances and sample sizes determine the information.

Worked Example: Adding Two-Parameter Contributions

Consider a hypothetical model with a two-component dimensionless parameter vector \(\theta=(\theta_1,\theta_2)^\mathsf{T}\). Source A supplies five independent normal measurements with known unit variance. At the nominal parameter value, their mean sensitivities with respect to \((\theta_1,\theta_2)\) are \((1,1)\), \((1,0)\), \((1,0)\), \((1,0)\), and \((0,1)\). Source B independently supplies four such measurements, with sensitivities \((1,0)\), \((0,1)\), \((0,1)\), and \((0,1)\). For a normal observation with known unit variance and mean sensitivity vector \(g\), its information contribution is \(gg^\mathsf{T}\). The sources have the same parameterization and are independent by assumption.

For source A, adding the five outer products gives

\mathcal I_A= \begin{pmatrix}1&1\\1&1\end{pmatrix} +3\begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}0&0\\0&1\end{pmatrix} = \begin{pmatrix}4&1\\1&2\end{pmatrix}.

For source B, the one sensitivity \((1,0)\) contributes one unit in the first diagonal position, while the three \((0,1)\) sensitivities contribute three units in the second:

\mathcal I_B= \begin{pmatrix}1&0\\0&3\end{pmatrix}, \qquad \mathcal I_{\mathrm{total}}=\mathcal I_A+\mathcal I_B =\begin{pmatrix}5&1\\1&5\end{pmatrix}.

The determinant of the total matrix is \(5(5)-1(1)=24\), which is positive; its first leading diagonal entry is also positive, so the matrix is positive definite and invertible. The inverse is

\mathcal I_{\mathrm{total}}^{-1} =\frac{1}{24}\begin{pmatrix}5&-1\\-1&5\end{pmatrix} =\begin{pmatrix}0.2083&-0.04167\\-0.04167&0.2083\end{pmatrix}.

Thus, under the regular large-sample approximation, the two parameter variances are each about 0.2083 in squared parameter units, and their covariance is about \(-0.04167\) in the product of the parameter units. The off-diagonal information and covariance terms matter: matrix addition preserves the joint information structure, while the inverse—not elementwise reciprocals—provides the covariance approximation.

Worked Example: Why Correlated Sources Cannot Be Added Naïvely

Suppose two hypothetical measurement types both have mean \(\theta\), measured in mg/L, and marginal variance \(1\;(\mathrm{mg/L})^2\). Their observed values are 1.2 and 0.8 mg/L. However, the measurements are correlated with known correlation \(0.5\), so their joint covariance matrix is

\Sigma=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}(\mathrm{mg/L})^2.

The joint model is multivariate normal with mean vector \(\theta\mathbf 1\). Its information for the scalar mean is \(\mathbf 1^\mathsf{T}\Sigma^{-1}\mathbf 1\), where \(\mathbf 1=(1,1)^\mathsf{T}\). Since

\Sigma^{-1}=\frac{1}{0.75}\begin{pmatrix}1&-0.5\\-0.5&1\end{pmatrix}, \qquad \mathbf 1^\mathsf{T}\Sigma^{-1}\mathbf 1=\frac{4}{3},

the joint information is \(1.333\;(\mathrm{mg/L})^{-2}\), not 2.000. Adding the marginal information \(1+1\) would incorrectly assume independence and overstate the information. The observed values do not alter this expected-information result. The correct calculation uses the joint likelihood, which retains the covariance between measurements.

Practical Checks, Alternatives, and Limits

Before adding contributions, check that every matrix has the same row and column ordering, parameter scale, and dimension. Information depends on parameterization, so matrices expressed for different transformations cannot be added as if their entries represented the same derivatives. Also verify that each source's likelihood conditions and variance assumptions are appropriate. If an observation type provides no information on a parameter, its corresponding sensitivity is zero; if a source has its own parameters, represent those explicitly rather than silently equating them with parameters from another source.

When the independent-factor assumption is doubtful, the main alternative is to specify and analyze the joint likelihood directly. If sources share participants, samples, latent effects, or processing steps, their dependence may create nonzero cross-information terms. A robust or composite-likelihood analysis may be appropriate in some settings, but its uncertainty calculation is not obtained simply by inverting a naïvely summed matrix. The likelihood and covariance assumptions must match the data-generating structure.

A singular total information matrix means at least one parameter direction is not locally distinguished by the combined observations. Adding information can increase rank, but it does not guarantee that all parameters become identifiable. Even when the matrix is invertible, a very small eigenvalue indicates weak information in some direction and can make the inverse unstable. Check symmetry, eigenvalues or determinant, and the condition of the matrix; also verify units and numerical scaling before interpreting an inverse.

A useful matrix-level fact is that adding a positive semidefinite contribution cannot reduce information in the Loewner ordering: \(\mathcal I_A+\mathcal I_B\succeq\mathcal I_A\). When both matrices are positive definite, inversion reverses this ordering, so \((\mathcal I_A+\mathcal I_B)^{-1}\preceq\mathcal I_A^{-1}\). This says the covariance approximation cannot increase in any quadratic parameter direction under the model. It does not mean every off-diagonal covariance must move toward zero, nor does it address model misspecification or dependence that was omitted.

Common pitfall: “Each source has its own information matrix, so I can always add them.” This is false if the sources are dependent, use incompatible parameterizations, or have been constructed from overlapping observations. First establish that the joint likelihood factors and that the matrices refer to the same parameter vector. If not, derive information from the joint model instead.

Check Your Understanding

Work through each question, then compare with the solutions.

Two independent sources provide information 2.5 and 1.5 for the same scalar parameter. What is the total information, and what is its inverse-information variance approximation?
Why do the cross-information terms vanish when independent sources have mean-zero scores?
Two sources have information matrices in different parameterizations. May their matrices be added directly? Explain.
A source contributes a positive semidefinite matrix to an already positive definite total information matrix. What happens to the inverse-information covariance ordering?

Solutions. (1) The total is \(2.5+1.5=4.0\); the inverse-information variance approximation is \(1/4.0=0.25\) in squared parameter units. (2) Independence gives \(E[U_1U_2^\mathsf{T}]=E[U_1]E[U_2]^\mathsf{T}=0\), because both score means are zero. (3) No. Re-express the contributions in the same parameterization, using the appropriate information transformation, before adding them. (4) The total information increases in the positive semidefinite ordering, so its inverse decreases in that ordering; covariance cannot increase in any quadratic parameter direction under the stated model assumptions.

Adding Independent Information Contributions

What You'll Learn