How Information Accumulates Across Subjects

Why Subject Counts Matter for Information

In a population study, each subject provides data about shared population parameters. If subjects are independent under the model, their likelihood contributions combine, and their information can accumulate. The key consequence is that adding subjects generally improves precision. But the size and direction of that improvement depend on the information each subject contributes: a cohort is not necessarily equivalent to repeating one design many times.

The result applies to a common parameter vector and independent subject-level data contributions. A subject’s contribution can summarize all of that subject’s measurements; independence is required across subjects, not among measurements within a subject. Within-subject dependence must already be represented in the subject-level likelihood.

Definition: Let \(Y_i\) denote all observed data from subject \(i\), with density or probability mass function \(f_i(Y_i\mid\boldsymbol{\theta})\) under a shared parameter vector \(\boldsymbol{\theta}\). The subject-level score is \(U_i(\boldsymbol{\theta})=\partial\log f_i(Y_i\mid\boldsymbol{\theta})/\partial\boldsymbol{\theta}\). Subject \(i\)'s expected Fisher information matrix is \(\mathcal I_i(\boldsymbol{\theta})=E[U_iU_i^\mathsf{T}]\). Under standard regularity conditions, it also equals \(-E[\partial^2\log f_i/\partial\boldsymbol{\theta}\partial\boldsymbol{\theta}^\mathsf{T}]\).

The regularity conditions include a correctly specified differentiable likelihood, a parameter value in the interior of the parameter space, and enough integrability to interchange differentiation and expectation. In particular, the score has mean zero, \(E[U_i]=\boldsymbol{0}\), and the expected negative Hessian agrees with the score-variance definition. When the model has boundary parameters or other irregular features, these identities or the usual normal approximation may require special care.

Why Independent Subjects’ Matrices Add

If subjects are independent, the joint likelihood factors into subject-level likelihoods. Taking logarithms turns this product into a sum, so the joint score is the sum of the subject-level scores. The expected information then adds because the cross-products between distinct subjects’ scores have expectation zero.

L(\boldsymbol{\theta})=\prod_{i=1}^{n}f_i(Y_i\mid\boldsymbol{\theta}), \qquad \ell(\boldsymbol{\theta})=\sum_{i=1}^{n}\ell_i(\boldsymbol{\theta}), \qquad U(\boldsymbol{\theta})=\sum_{i=1}^{n}U_i(\boldsymbol{\theta}).

\mathcal I_{\mathrm{total}}(\boldsymbol{\theta}) =E[UU^\mathsf{T}] =\sum_{i=1}^{n}\mathcal I_i(\boldsymbol{\theta}) +\sum_{i\ne j}E[U_iU_j^\mathsf{T}] =\sum_{i=1}^{n}\mathcal I_i(\boldsymbol{\theta}).

For distinct subjects, independence gives \(E[U_iU_j^\mathsf{T}]=E[U_i]E[U_j]^\mathsf{T}=\boldsymbol{0}\). Equivalently, the observed negative Hessian of the joint log likelihood is the sum of subject-level negative Hessians, and taking expectations gives the same result. Thus the sum rule is a property of the joint likelihood, not a claim that every subject has an identical contribution.

Key result: For independent subjects sharing \(\boldsymbol{\theta}\), total expected Fisher information is the matrix sum \(\mathcal I_{\mathrm{total}}=\sum_i\mathcal I_i\). Only when all subjects have the same information matrix \(\mathcal I_{\mathrm{subject}}\) does this simplify to \(\mathcal I_{\mathrm{total}}=n\mathcal I_{\mathrm{subject}}\).

The matrix sum adds information about all parameter directions, including cross-information. It is not generally valid to add only the diagonal entries and ignore off-diagonal entries. A positive off-diagonal entry, for example, indicates that the score components for two parameters tend to vary together; it affects the inverse total matrix and therefore the uncertainty of each estimated parameter.

From Total Information to Standard Errors

Under the usual large-sample likelihood approximation, the estimator \(\hat{\boldsymbol{\theta}}\) is approximately multivariate normal around the true parameter, with covariance matrix given by the inverse total expected information. In applications, the information is often evaluated at an estimate or at nominal parameter values. For parameter \(j\), the approximate standard error is the square root of the \(j\)th diagonal entry of the inverse matrix.

\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\theta}}) \ \approx\ \mathcal I_{\mathrm{total}}(\boldsymbol{\theta})^{-1}, \qquad SE(\hat{\theta}_j) \ \approx\ \sqrt{\left[\mathcal I_{\mathrm{total}}(\boldsymbol{\theta})^{-1}\right]_{jj}}.

For a single scalar parameter, if all \(n\) subjects have the same information \(I_{\mathrm{subject}}\), then \(I_{\mathrm{total}}=nI_{\mathrm{subject}}\), and the approximate standard error is \(1/\sqrt{nI_{\mathrm{subject}}}\). Doubling the cohort size therefore multiplies the standard error by \(1/\sqrt{2}\), not by one half. This familiar square-root scaling also holds for a vector parameter under identical subject designs: \((n\mathcal I_{\mathrm{subject}})^{-1}=\mathcal I_{\mathrm{subject}}^{-1}/n\), so each standard error scales by \(1/\sqrt{n}\), provided the information matrix is invertible and the same model and design are maintained.

Worked Examples

Worked Example: Cohort-Size Scaling for a Scalar Parameter

Suppose, hypothetically, that each subject contributes one measurement \(Y_i\sim N(\theta,16)\), independently, where the variance \(16\ \text{units}^2\) is known and the common mean \(\theta\) is the parameter of interest. Find the information and approximate standard error for \(n=25\), then for \(n=100\).

For one observation, the log likelihood, up to terms not involving \(\theta\), is \(\ell_i(\theta)=-(Y_i-\theta)^2/32\). Its score is \(U_i=(Y_i-\theta)/16\). Since \(\operatorname{Var}(Y_i)=16\), the subject information is \(E[U_i^2]=16/16^2=1/16\ \text{units}^{-2}\).

For 25 independent subjects, information adds:

I_{\mathrm{total}}=25\left(\frac{1}{16}\right)=1.5625\ \text{units}^{-2}, \qquad SE(\hat{\theta})\approx \frac{1}{\sqrt{1.5625}}=0.8000\ \text{units}.

For 100 subjects, \(I_{\mathrm{total}}=100/16=6.250\ \text{units}^{-2}\), giving \(SE(\hat{\theta})\approx 1/\sqrt{6.250}=0.4000\ \text{units}\). The cohort grew by a factor of four while the standard error halved, consistent with the \(1/\sqrt{n}\) relationship. In this normal model with known variance, the variance of the sample mean is exactly \(16/n\), so this standard-error calculation is exact as well as the corresponding large-sample information calculation.

Worked Example: Unequal Subject Contributions

Consider a hypothetical scalar-parameter model \(Y_i\sim N(x_i\theta,4)\), with known variance \(4\ \text{units}^2\). The fixed, known design values for three independent subjects are \(x_1=1\), \(x_2=1\), and \(x_3=2\). Calculate total information and the approximate standard error for \(\theta\).

For a subject with design value \(x_i\), the score is \(x_i(Y_i-x_i\theta)/4\). Its variance, and hence its information, is \(x_i^2/4\). Therefore the three contributions are \(1/4\), \(1/4\), and \(4/4\), in the information units corresponding to \(\theta\).

I_{\mathrm{total}}=\sum_{i=1}^{3}\frac{x_i^2}{4} =\frac{1^2+1^2+2^2}{4} =\frac{6}{4}=1.500, \qquad SE(\hat{\theta})\approx \frac{1}{\sqrt{1.500}}=0.8165.

The subjects do not contribute equally: the subject with \(x_i=2\) contributes four times as much information as either subject with \(x_i=1\). Multiplying the first subject’s information by three would give \(3/4=0.750\), only half the actual total. The sum is appropriate because the independent subjects have different known design values; a cohort-size multiplier is appropriate only for repeated identical contributions.

Worked Example: Inverting the Total Matrix

Suppose a hypothetical study estimates two mean-model parameters, \(\boldsymbol{\theta}=(\theta_0,\theta_1)^\mathsf{T}\). Each independent subject contributes one observation \(Y_i\sim N(\theta_0+x_i\theta_1,1)\), with known residual variance \(1\ \text{unit}^2\). Six subjects have \(x_i=0\), and four have \(x_i=1\). Find the total information matrix and approximate standard errors.

For a subject with design row \((1,x_i)\), the information contribution is the outer product of that row with itself, divided by the variance. Thus an \(x_i=0\) subject contributes \(\begin{pmatrix}1&0\\0&0\end{pmatrix}\), while an \(x_i=1\) subject contributes \(\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Summing six and four contributions gives:

\mathcal I_{\mathrm{total}} =6\begin{pmatrix}1&0\\0&0\end{pmatrix} +4\begin{pmatrix}1&1\\1&1\end{pmatrix} =\begin{pmatrix}10&4\\4&4\end{pmatrix}.

The determinant is \(10(4)-4(4)=24\), which is positive, so the matrix is invertible. Its inverse is:

\mathcal I_{\mathrm{total}}^{-1} =\frac{1}{24}\begin{pmatrix}4&-4\\-4&10\end{pmatrix} =\begin{pmatrix}0.1667&-0.1667\\-0.1667&0.4167\end{pmatrix}.

Taking square roots of the diagonal entries gives \(SE(\hat{\theta}_0)\approx\sqrt{0.1667}=0.4082\) units and \(SE(\hat{\theta}_1)\approx\sqrt{0.4167}=0.6455\) units. The negative off-diagonal entry in the inverse indicates negative covariance in this approximation; it does not make either variance negative. The slope parameter is less precisely estimated because the \(x=0\) subjects provide no direct slope information.

Short Derivation of the Square-Root Rule

The scaling rule follows directly from the matrix covariance approximation. Suppose every subject has the same nonsingular information matrix \(\mathcal I_0\). Additivity gives \(\mathcal I_{\mathrm{total}}=n\mathcal I_0\). For any positive scalar \(n\), \((n\mathcal I_0)^{-1}=(1/n)\mathcal I_0^{-1}\), because multiplying these matrices gives the identity. Therefore every diagonal variance in the approximation is divided by \(n\), and its standard deviation is divided by \(\sqrt{n}\).

\operatorname{Var}(\hat{\theta}_j)\approx \frac{[\mathcal I_0^{-1}]_{jj}}{n} \quad\Longrightarrow\quad SE(\hat{\theta}_j)\approx \frac{\sqrt{[\mathcal I_0^{-1}]_{jj}}}{\sqrt{n}}.

This argument requires the same per-subject information and a nonsingular matrix. With heterogeneous designs, total information remains a sum, but it need not equal \(n\) times a common matrix. Precision still follows from the inverse of that sum, not from cohort size alone.

Diagnostics, Limitations, and Common Mistakes

Before interpreting a total information matrix, check whether the assumptions behind additivity and the standard-error approximation are credible. The likelihood should factor across subjects, the parameterization should be consistent across contributions, and the matrix should contain information for all parameters whose uncertainty is being reported. A singular or nearly singular total matrix cannot support stable standard errors for every parameter. Sparse contributions in an important parameter direction can also make uncertainty large even when the cohort is numerically substantial.

A common mistake is to see \(n\) subjects and assume total information must be \(n\) times the information of any one subject. The correction is to sum the actual subject-level matrices. The multiplier formula applies only to identical contributions. Another mistake is to calculate standard errors as reciprocal square roots of the diagonal entries of the information matrix. That works for an isolated scalar parameter, but with multiple parameters the correct calculation uses the diagonal of the inverse full matrix. Cross-information can change the result.

For shared nuisance parameters, first add the full information blocks across subjects. If the target parameter is then adjusted for nuisance parameters by a Schur complement, form that complement from the total blocks; in general, subject-level adjusted complements do not simply add. This distinction matters because subjects jointly estimate common nuisance parameters as well as target parameters.

The sum rule also depends on independence. If subject records are dependent, the joint score still may be written as a sum under an appropriate likelihood, but cross-subject score covariances need not vanish. The total information then includes those cross terms, and the independent-subject formula is not justified without further modeling. Finally, the inverse Fisher matrix is a model-based approximation: misspecification, boundary parameters, small samples, or weak identification can make reported standard errors misleading even when the arithmetic is correct.

Key takeaway: Independent subjects contribute additive expected Fisher matrices. Cohort size yields the simple \(1/\sqrt{n}\) standard-error rule only when per-subject information is identical and the relevant total information is nonsingular; otherwise, sum the actual matrices and invert the result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Three independent subjects have scalar information contributions \(0.2\), \(0.3\), and \(0.5\). What is total information, and what is the approximate standard error?
If identical independent subjects contribute information \(\mathcal I_0\), how does the approximate standard error change when sample size increases from \(n\) to \(9n\)?
Why is the inverse of a total two-parameter information matrix needed to obtain standard errors, rather than taking reciprocal square roots of its diagonal entries?
What additional issue arises if subject scores are dependent across subjects?

Solutions. 1. Add the contributions: \(I_{\mathrm{total}}=0.2+0.3+0.5=1.0\). The approximate standard error is \(1/\sqrt{1.0}=1.0\) in the parameter’s units. 2. Information is multiplied by nine, so variance is divided by nine and standard error is divided by three. 3. Off-diagonal information represents cross-information between parameters; it affects the variances after inversion. The standard errors are square roots of the diagonal entries of the inverse full matrix. 4. Cross-subject score covariances may be nonzero, so the total information includes cross terms and need not equal the sum of individual expected information matrices.