Information Growth and Sample Size

Why Sample Size Is Not the Whole Story

Increasing the number of observations usually improves precision, but the count \(n\) alone does not determine how much an estimator improves. Each observation contributes information about the parameters, and the contribution can vary with the model and with the covariates. If the observations are informative in all parameter directions, total information typically grows in proportion to \(n\), giving the familiar root-\(n\) scale for estimation error. If an important predictor barely varies, or its values are distributed very unevenly, information about some coefficients can grow much more slowly.

The central distinction is between sample size and information in a particular parameter direction. The former counts observations; the latter measures how sharply the likelihood distinguishes nearby parameter values. This tutorial makes that distinction precise and shows how to use it to understand estimator precision.

Information Accumulates Across Observations

Let \(Y_1,\ldots,Y_n\) be conditionally independent responses in a correctly specified GLM, with covariates \(x_i\) treated as given for this calculation. Write \(\ell_n(\beta)\) for the log-likelihood for the regression parameter \(\beta\), and let \(U_i(\beta)\) be the score contribution from observation \(i\). Under the usual differentiability and integrability conditions that allow differentiation under the integral, each score has conditional mean zero. The expected Fisher information is the covariance matrix of the score, equivalently the negative expected Hessian when the information identity applies.

Definition: The expected Fisher information in the sample is \( \mathcal I_n(\beta)=E_\beta[U_n(\beta)U_n(\beta)^T] \), where \(U_n(\beta)=\nabla_\beta\ell_n(\beta)\). For conditionally independent observations with mean-zero scores, it is the sum of the individual expected information matrices. It describes local likelihood curvature: larger information in a direction generally means greater precision in that direction.

For a GLM with linear predictor \(\eta_i=x_i^T\beta\), conditional mean \(\mu_i\), variance \(\operatorname{Var}(Y_i\mid x_i)=\phi V(\mu_i)\), and link derivative \(d\mu_i/d\eta_i\), the usual expected information for \(\beta\), with dispersion \(\phi\) treated as known, is

\mathcal I_n(\beta) = \sum_{i=1}^n \frac{1}{\phi V(\mu_i)} \left(\frac{d\mu_i}{d\eta_i}\right)^2 x_i x_i^T .

Thus, the contribution of an observation depends both on its covariate vector and on its model-based weight. A large \(n\) guarantees substantial information in every direction only if the covariate patterns and weights collectively support every parameter direction.

Result: For conditionally independent observations with differentiable likelihoods, finite score second moments, and valid differentiation under the integral, expected information adds: \( \mathcal I_n(\beta)=\sum_{i=1}^n\mathcal I_i(\beta) \).

To see this, write \(U_n=\sum_i U_i\). Conditional independence gives \(E_\beta[U_iU_j^T\mid x_1,\ldots,x_n]=E_\beta[U_i\mid x_i]E_\beta[U_j^T\mid x_j]=0\) for \(i\ne j\), because each score has mean zero. Expanding the score covariance therefore leaves only the diagonal terms:

E_\beta[U_nU_n^T] = \sum_{i=1}^n E_\beta[U_iU_i^T] + \sum_{i\ne j}E_\beta[U_iU_j^T] = \sum_{i=1}^n\mathcal I_i(\beta).

The cancellation of cross terms is what makes information additive; it does not say that every observation contributes the same amount.

Information Growth and Root-n Precision

For a fixed-dimensional parameter, a standard large-sample result says that the maximum-likelihood estimator is asymptotically normal when the model is regular and the information is sufficiently well behaved. One common formulation is \( \mathcal I_n(\beta_0)^{1/2}(\hat\beta-\beta_0)\xrightarrow{d}N(0,I) \). This information-scaled statement allows different parameter directions to accumulate information at different rates. The familiar root-\(n\) result follows when the information per observation approaches a positive-definite limit.

Theorem: Suppose the parameter dimension is fixed, \(\mathcal I_n(\beta_0)\) is positive definite with probability tending to one, and \( \mathcal I_n(\beta_0)^{1/2}(\hat\beta-\beta_0)\xrightarrow{d}N(0,I) \). If \( \mathcal I_n(\beta_0)/n\xrightarrow{p}I_0 \), where \(I_0\) is a finite positive-definite matrix, then \( \sqrt n(\hat\beta-\beta_0)\xrightarrow{d}N(0,I_0^{-1}) \).

The information-scaled normality in the theorem is the usual regular-likelihood asymptotic result; the additional information-growth condition specifies when its scaling can be expressed using \(n\). To prove the conclusion, set \(Z_n=\mathcal I_n(\beta_0)^{1/2}(\hat\beta-\beta_0)\). Then

\sqrt n(\hat\beta-\beta_0) = \left(\frac{\mathcal I_n(\beta_0)}{n}\right)^{-1/2}Z_n.

The map taking a positive-definite matrix to its inverse square root is continuous. Hence the first factor converges in probability to \(I_0^{-1/2}\), while \(Z_n\) converges in distribution to \(N(0,I)\). Slutsky's theorem gives a limiting normal distribution with covariance \(I_0^{-1/2}I I_0^{-1/2}=I_0^{-1}\). This proves the claim. \(\square\)

In practical terms, if a scalar parameter has information approximately \(cn\), for some \(c>0\), its approximate standard error is \(1/\sqrt{cn}\). Multiplying the sample size by four then approximately halves that standard error. In a model with several coefficients, root-\(n\) precision in every direction requires the information matrix per observation to remain nonsingular, not merely that its entries grow.

Worked Examples: How Covariates Change Information

Worked Example: A Bernoulli Intercept

Suppose \(Y_1,\ldots,Y_n\) are independent Bernoulli observations with common probability \(p\), and the parameter is the logit \(\beta=\log\{p/(1-p)\}\). Find the information at \(p=0.4\) for \(n=100\) and \(n=400\), and compare the approximate standard errors.

For one observation, the log-likelihood is \(y\beta-\log(1+e^\beta)\). Its score is \(y-p\), whose variance is \(p(1-p)\). Thus the information per observation is \(p(1-p)=0.4(0.6)=0.24\), and additivity gives \(\mathcal I_n(\beta)=0.24n\).

At \(n=100\), the information is \(24\), so the information-based standard error is approximately \(1/\sqrt{24}=0.204\). At \(n=400\), it is \(96\), giving \(1/\sqrt{96}=0.102\). The sample size has quadrupled and the approximate standard error has halved, as expected under information proportional to \(n\).

Worked Example: Imbalance in a Binary Predictor

Consider logistic regression with an intercept and a binary predictor \(x_i\in\{0,1\}\). Suppose \(n=100\), the true coefficients are both zero, and therefore every response probability is \(0.5\). Compare a balanced design with 50 observations at each predictor value to a design with 90 at \(x=0\) and 10 at \(x=1\).

At probability \(0.5\), each observation has logistic-regression weight \(p_i(1-p_i)=0.25\). If \(m\) observations have \(x=1\) and \(n-m\) have \(x=0\), the information matrix for the intercept and slope is

\mathcal I_n = 0.25 \begin{pmatrix} n & m\\ m & m \end{pmatrix}.

Because the intercept is also estimated, the information available for the slope after accounting for the intercept is the Schur complement of its intercept block: \(0.25\{m-m^2/n\}=0.25m(n-m)/n\). For the balanced design this is \(0.25(50)(50)/100=6.25\), giving an approximate slope standard error \(1/\sqrt{6.25}=0.400\). For the 90/10 design it is \(0.25(10)(90)/100=2.25\), giving \(1/\sqrt{2.25}=0.667\).

Both designs have 100 observations and the same probabilities, but the imbalanced design contains less information for comparing the two predictor groups. There are fewer observations in the smaller group, so the slope is less precisely estimated.

Worked Example: Shrinking Predictor Variation

To see how weak variation can change the information growth rate, take even \(n\), a logistic model with intercept and slope, and covariate values \(x_i=+n^{-1/4}\) for half the observations and \(x_i=-n^{-1/4}\) for the other half. Let both true coefficients be zero, so all probabilities are \(0.5\). Find the information for the slope after accounting for the intercept.

The covariate mean is zero, and the sum of squared deviations is \( \sum_i(x_i-\bar x)^2=n(n^{-1/4})^2=n^{1/2} \). With logistic weight \(0.25\), the slope information after accounting for the intercept is therefore \(0.25n^{1/2}\), not a constant times \(n\). Its information-based standard error is \(1/\sqrt{0.25n^{1/2}}=2n^{-1/4}\).

For \(n=10{,}000\), this design gives slope information \(25\) and an approximate standard error \(0.2\). If instead the covariates were \(+1\) and \(-1\) in equal numbers, the information would be \(0.25n=2{,}500\), and the corresponding standard error would be \(0.02\). The sample sizes match, but the shrinking predictor spread makes the slope much harder to estimate. The information calculation shows why ordinary root-\(n\) scaling is not justified here; an information-based asymptotic approximation would require its own regularity conditions.

Reading Information in Multiple Dimensions

For a vector parameter, information can grow quickly for some combinations of coefficients and slowly for others. If \(\mathcal I_n\) has a small eigenvalue, there is a direction in parameter space along which the likelihood is relatively flat. Estimation is then imprecise in that direction, even if the total sample is large. A covariance approximation based on inverse information makes this distinction explicit: the variance of a contrast \(a^T\hat\beta\) is approximately \(a^T\mathcal I_n^{-1}a\).

This is why reporting only \(n\), or even only a single overall summary of the design, can be misleading. A large sample cannot compensate automatically for a predictor that barely changes, a category represented by very few observations, or strong redundancy among predictors. These patterns can make the smallest eigenvalue of information grow slowly and leave particular coefficient combinations uncertain.

Common Mistake: Treating Every Observation as Equally Informative

Wrong reasoning: “The sample size is \(n\), so every coefficient must have standard error of order \(n^{-1/2}\).” This silently assumes that information per observation remains substantial in every parameter direction. The examples show that the assumption can fail: imbalance reduces information for a group comparison, and a predictor whose spread shrinks with \(n\) may contribute information at a slower rate.

Correction: Examine the information matrix, or the information for the coefficient or contrast of interest. Root-\(n\) precision follows when information grows proportionally to \(n\) and its normalized limit is positive definite. If a direction has less information, its precision is worse; the appropriate rate must be assessed from that direction's information growth, together with the relevant asymptotic regularity conditions.

Key takeaway: More observations usually mean more information, but the rate and distribution of that information across parameter directions determine precision. Root-\(n\) behavior requires information proportional to sample size in every direction being estimated.

Check Your Understanding

Use information growth, rather than the observation count alone, to reason through these questions.

If scalar information is approximately \(5n\), what happens to the approximate standard error when \(n\) is multiplied by nine?
Why does a logistic-regression slope for a binary predictor depend on how many observations fall in each group?
In the shrinking-variation example, what is the order of the slope information as a function of \(n\), and what standard-error order does it suggest?
What condition on the limiting information matrix ensures root-\(n\) precision for all fixed-dimensional coefficient directions?
Can a large sample still provide weak information about a particular coefficient combination? Explain using the eigenvalues of the information matrix.