Information Scaling With Replication

Why Replication Changes Precision

In a model-informed study, repeating an observation can make parameter estimates more precise, but the gain is not generally proportional to the number of replicates when precision is measured by a standard error. The key distinction is that information accumulates linearly, while approximate standard errors typically decrease in proportion to the square root of the information. For a vector of parameters, this relationship is expressed through the Fisher information matrix and its inverse.

The result is useful for comparing designs that repeat the same measurement under the same conditions. Its assumptions matter: independent replicates contribute additively, but correlated measurements cannot automatically be counted as separate independent contributions. Also, replication of an identical design is a special case; observations made under different conditions may contribute different amounts and directions of information.

Information From Independent Replicates

Let \(\theta\) be a \(p\)-dimensional parameter vector. For one replicate with data \(Y\), write its log-likelihood as \(\ell_1(\theta;Y)\), its score as \(U_1(\theta)=\partial \ell_1(\theta;Y)/\partial\theta\), and its expected Fisher information as \(\mathcal{I}_1(\theta)=E_\theta[U_1(\theta)U_1(\theta)^\mathsf{T}]\). Under the usual regularity conditions for the score and likelihood, this also equals the negative expected Hessian, \(-E_\theta[\partial^2\ell_1(\theta;Y)/\partial\theta\partial\theta^\mathsf{T}]\).

Definition: Independent replicates are data contributions that are mutually independent conditional on the model parameters. If replicate \(i\) has log-likelihood \(\ell_i(\theta)\), its observed information is \(\mathcal{J}_i(\theta)=-\partial^2\ell_i(\theta)/\partial\theta\partial\theta^\mathsf{T}\), and its expected information is \(\mathcal{I}_i(\theta)=E_\theta[\mathcal{J}_i(\theta)]\), when the expectation and derivatives are well-defined.

Independence makes the joint likelihood a product of replicate likelihoods, so the joint log-likelihood is a sum. Differentiating that sum shows that observed information adds. Expected information therefore adds as well, by taking expectations of the sum. Equivalently, under the regularity condition that each score has expectation zero, the cross-products between independent scores have expectation zero.

\ell(\theta)=\sum_{i=1}^{n}\ell_i(\theta), \qquad \mathcal{J}(\theta)=\sum_{i=1}^{n}\mathcal{J}_i(\theta), \qquad \mathcal{I}(\theta)=\sum_{i=1}^{n}\mathcal{I}_i(\theta).

If the replicates are identically distributed and use the same measurement conditions, then every replicate has the same expected information matrix, \(\mathcal{I}_1(\theta)\). In that case, the expected information for \(n\) replicates is \(n\mathcal{I}_1(\theta)\). The same exact scaling applies to observed information only when the observed information contributions themselves are identical; in ordinary random data, their realized values can differ even when their expectations are equal.

Key result: For \(n\) independent, identically distributed replicates with per-replicate expected information \(\mathcal{I}_1(\theta)\), the total expected information is \(\mathcal{I}_n(\theta)=n\mathcal{I}_1(\theta)\). This requires the same model, parameter value, and replicate design, as well as the regularity conditions needed to define the information.

Why Approximate Standard Errors Scale Differently

When the information matrix is nonsingular and the estimator has a suitable large-sample normal approximation, the approximate covariance matrix of the maximum likelihood estimator is \(\mathcal{I}_n(\theta)^{-1}\). This is a covariance approximation, not a guarantee of exact finite-sample uncertainty. For identical independent replicates, multiplying the information matrix by \(n\) divides its inverse by \(n\):

\mathcal{I}_n(\theta)^{-1} = \bigl(n\mathcal{I}_1(\theta)\bigr)^{-1} = \frac{1}{n}\mathcal{I}_1(\theta)^{-1}.

An approximate standard error is the square root of a diagonal element of this covariance matrix. Thus, under these conditions, each approximate standard error is multiplied by \(1/\sqrt{n}\) relative to the one-replicate approximation. Four times as many identical independent replicates gives half the approximate standard error—not one quarter. The result applies to the matrix as a whole; it does not say that every parameter is estimated equally precisely.

The derivation depends on invertibility: if \(\mathcal{I}_1(\theta)\) has no inverse, the displayed covariance formula does not apply as written. Invertibility alone is also not sufficient for a good approximation; the estimator must be in a regime where the likelihood-based normal approximation is credible.

Worked Examples

Worked Example: Replicating a Measurement of a Mean

Suppose, hypothetically, that each independent replicate is one concentration measurement \(Y_i\) from a normal model \(Y_i\sim N(\mu,\sigma^2)\), where \(\sigma=4\) mg/L is known and \(\mu\) is the mean concentration in mg/L. The measurements are independent and share the same distribution. Find the information and approximate standard error for \(\mu\) with \(n=16\) replicates.

For one observation, the log-likelihood, apart from a constant, is \(\ell_1(\mu)=-(Y_i-\mu)^2/(2\sigma^2)\). Its score is \((Y_i-\mu)/\sigma^2\), so its expected information is \(1/\sigma^2\). Substituting the known standard deviation gives:

\mathcal{I}_1(\mu)=\frac{1}{(4\ \mathrm{mg/L})^2} =0.0625\ (\mathrm{mg/L})^{-2}.

The 16 independent observations have total information \(16(0.0625)=1.00\ (\mathrm{mg/L})^{-2}\). The inverse information is \(1.00\ (\mathrm{mg/L})^2\), so the approximate standard error is \(\sqrt{1.00}=1.00\) mg/L. Here the normal known-variance model also gives the exact standard error of the sample mean, \(4/\sqrt{16}=1\) mg/L. The assumptions are explicit: the observations are independent, have a common mean and known common variance, and follow the stated normal model.

Worked Example: Scaling a Two-Parameter Information Matrix

Suppose, hypothetically, that one independent replicate under a fixed design has expected information \[ \mathcal{I}_1= \begin{pmatrix} 4&1\\ 1&2 \end{pmatrix}, \] for two dimensionless parameters \(\theta_1\) and \(\theta_2\). Assume the 25 replicates are independent and have this same information matrix. Calculate the total information and the approximate standard errors.

The determinant of the single-replicate matrix is \(4(2)-1(1)=7\), so it is invertible. The total expected information is:

\mathcal{I}_{25}=25\mathcal{I}_1 = \begin{pmatrix} 100&25\\ 25&50 \end{pmatrix}.

The single-replicate inverse is \(\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}\). Dividing it by 25 gives the approximate covariance matrix:

\mathcal{I}_{25}^{-1} = \frac{1}{175} \begin{pmatrix} 2&-1\\ -1&4 \end{pmatrix} = \begin{pmatrix} 0.01143&-0.005714\\ -0.005714&0.02286 \end{pmatrix}.

The approximate standard error for \(\theta_1\) is \(\sqrt{0.01143}=0.1069\), and for \(\theta_2\) it is \(\sqrt{0.02286}=0.1512\), both in the respective parameter units (dimensionless here). These are \(1/5\) of the single-replicate approximations, as expected because \(\sqrt{25}=5\). The matrix calculation requires identical independent replicate designs and a nonsingular information matrix; the standard-error interpretation additionally relies on an adequate large-sample approximation.

Worked Example: Replicates With Two Different Designs

Now suppose, hypothetically, that 8 independent subjects follow design A, each contributing expected information \[ \mathcal{I}_A= \begin{pmatrix} 2&0\\ 0&1 \end{pmatrix}, \] and 12 independent subjects follow design B, each contributing \[ \mathcal{I}_B= \begin{pmatrix} 1&0.5\\ 0.5&2 \end{pmatrix}. \] Assume both matrices are evaluated for the same two dimensionless parameters under the same model. Find the total information and approximate standard errors.

The designs are not identical, so there is no single per-subject matrix to multiply by 20. Instead, add the contributions for each design:

\mathcal{I}_{\mathrm{total}} =8\mathcal{I}_A+12\mathcal{I}_B = \begin{pmatrix} 16&0\\ 0&8 \end{pmatrix} + \begin{pmatrix} 12&6\\ 6&24 \end{pmatrix} = \begin{pmatrix} 28&6\\ 6&32 \end{pmatrix}.

Its determinant is \(28(32)-6^2=860\), which is positive, so this matrix is invertible. Its inverse is:

\mathcal{I}_{\mathrm{total}}^{-1} = \frac{1}{860} \begin{pmatrix} 32&-6\\ -6&28 \end{pmatrix}.

The approximate standard errors are \(\sqrt{32/860}=0.1929\) for \(\theta_1\) and \(\sqrt{28/860}=0.1804\) for \(\theta_2\). The calculation uses independence across subjects and the specified per-subject matrices. It does not assume that all subjects have the same design. This example illustrates the general additive rule: add the information matrices for the actual independent contributions before taking the inverse.

Justification and Practical Checks

The matrix scaling result follows directly from the likelihood factorization. Under independence, the joint density is the product of the replicate densities. Taking logarithms converts that product to a sum, and taking two derivatives preserves the sum. Taking the expectation then gives the expected-information identity. The score-based argument gives the same result and clarifies why independence is important: for the total score \(U=\sum_i U_i\), its expected outer product contains cross-terms \(E_\theta[U_iU_j^\mathsf{T}]\). For \(i\ne j\), independence and zero-mean scores imply that these cross-terms are zero.

E_\theta[U_iU_j^\mathsf{T}] = E_\theta[U_i]E_\theta[U_j]^\mathsf{T} =0, \qquad i\ne j.

Before using a replication calculation, check the unit of replication and the structure of the data. In a pharmacometric setting, observations from the same individual may share subject-specific effects or other sources of dependence. Such observations should not be treated as independent replicates merely because they are separate measurements. The appropriate likelihood may still have independent contributions across subjects, but dependence within each subject must be represented in that subject’s contribution.

Check independence at the claimed replication level. If replicates share an unmodelled random effect, the independent-contribution argument does not apply to them as separate units.
Check whether the designs are actually identical. Different doses, sampling times, or observation conditions may produce different information matrices. Add those matrices individually rather than multiplying one of them by the total count.
Check matrix invertibility and scale. The inverse-information covariance approximation requires an invertible matrix. A very small eigenvalue can also signal that the resulting uncertainty approximation is highly sensitive or large.
Check the approximation, not just the arithmetic. Fisher information scaling is an information result; the standard-error formula is an asymptotic approximation. Small samples, weakly informed parameters, or model misspecification can make it inaccurate.

A common mistake is to say, “There are four times as many independent replicates, so the standard error is four times smaller.” The information does increase fourfold when the replicates are identical, but the covariance is inversely proportional to information and the standard error is the square root of variance. The correct change is a factor of \(1/\sqrt{4}=1/2\). Another mistake is to apply \(n\mathcal{I}_1\) to observations that are correlated or collected under different designs. In those cases, the independence or identical-design condition is not satisfied; use the correctly specified joint likelihood or add the distinct independent contributions instead.

When the Scaling Rule Is Useful—and Its Limits

For planning, the rule gives a quick benchmark: under identical independent replication and a reliable large-sample approximation, multiplying the sample size by \(c\) multiplies expected information by \(c\) and divides approximate standard errors by \(\sqrt{c}\). The inverse relationship helps explain why precision gains become progressively less pronounced as sample size grows. For instance, reducing an approximate standard error by half requires four times as many identical replicates, not twice as many.

This benchmark is not a complete design comparison. It assumes replication at the same design and does not determine whether observations at other conditions would be more informative for the parameter of interest. Nor does it guarantee that the model is correct, that all parameters are well identified, or that finite-sample uncertainty will match the approximation. If the contributions differ, add their expected information matrices; if dependence is present, derive information from the joint likelihood rather than counting observations as independent.

Check Your Understanding

Use the definitions and assumptions above. Solutions are included for checking your reasoning.

If 36 identical independent replicates provide information \(2.5\) per replicate for a scalar parameter, what are the total information and the multiplier on its approximate standard error relative to one replicate?
Why does expected information add for independent replicates, and what condition makes the score cross-terms vanish?
Two independent design groups contribute \(5\mathcal{I}_A\) and \(7\mathcal{I}_B\). When is it valid to replace their total with \(12\mathcal{I}_A\)?
Four measurements come from each of 10 subjects. What must be considered before treating all 40 measurements as independent replicates?

Solutions. (1) Total information is \(36(2.5)=90\). The approximate standard-error multiplier is \(1/\sqrt{36}=1/6\). (2) The joint log-likelihood is a sum for independent data, so its expected information is the sum of the individual expected information matrices. In the score argument, distinct scores are independent and have expectation zero, giving \(E_\theta[U_iU_j^\mathsf{T}]=0\) for \(i\ne j\). (3) Only when the two per-replicate information matrices are equal, \(\mathcal{I}_B=\mathcal{I}_A\), so their sum is \(5\mathcal{I}_A+7\mathcal{I}_A=12\mathcal{I}_A\). (4) Consider within-subject dependence, including shared subject effects, and specify the joint likelihood for each subject. Measurements within a subject are not automatically independent replicates; the independent units may instead be the 10 subjects.