Fixed-Design and Random-Design Information Limits

Why the Covariate Design Matters

In a regression model, information about the coefficients comes not only from the response distribution but also from the covariate values at which responses are observed. A large sample cannot compensate for a design that provides little variation in a direction needed to distinguish coefficients. To describe large-sample information precisely, we must say whether the covariates are treated as fixed or sampled randomly.

The distinction is especially clear in binary logistic regression. We will calculate the information per observation for both designs and show how its limit is determined by the empirical distribution of fixed covariates or by the population distribution of random covariates. Throughout, the regression model is correctly specified and the coefficient vector is fixed.

Conditional Information in Logistic Regression

Let \(Y_i\) be a binary response and \(x_i\) a scalar covariate. Write \(z_i=(1,x_i)^\mathsf{T}\), and let \(\beta=(\beta_0,\beta_1)^\mathsf{T}\). Under the logistic model, \(Y_i\mid x_i\) is Bernoulli with success probability \(p_\beta(x_i)\), where \(p_\beta(x)=\exp(\beta_0+\beta_1x)/(1+\exp(\beta_0+\beta_1x))\). Assume responses are conditionally independent given the covariates.

To see the formula, the contribution to the log-likelihood is \(\ell_i(\beta)=Y_i\log p_\beta(x_i)+(1-Y_i)\log\{1-p_\beta(x_i)\}\). Differentiating with respect to \(\beta\) gives the score \((Y_i-p_\beta(x_i))z_i\). Conditional on \(x_i\), the score has mean zero and covariance \(p_\beta(x_i)\{1-p_\beta(x_i)\}z_i z_i^\mathsf{T}\), since the variance of a Bernoulli variable with success probability \(p\) is \(p(1-p)\). This is the stated information contribution.

Fixed-Design Information Limits

In a fixed-design analysis, the covariates \(x_1,\ldots,x_n\) are treated as deterministic. Their empirical distribution is \(H_n=n^{-1}\sum_{i=1}^n\delta_{x_i}\), where \(\delta_x\) places all its mass at \(x\). The normalized information can then be written as an average with respect to that empirical distribution.

Here weak convergence means that integrals of every bounded continuous function converge. For compactly supported designs, every entry of the integrand is bounded and continuous, so this definition gives a direct route to the limit.

Proof. The \((r,s)\) entry of the normalized information is the empirical average of the corresponding entry of \(g_\beta(x)=w_\beta(x)(1,x)^\mathsf{T}(1,x)\). The logistic probability is continuous in \(x\), so \(w_\beta(x)\) is continuous; each entry of \(g_\beta\) is therefore continuous. On the common compact set, these entries are bounded. By weak convergence, the empirical integral of each entry converges to its integral under \(H\). Applying this argument to all four entries proves matrix convergence to \(\int g_\beta(x)\,dH(x)\). \(\square\)

Random-Design Information Limits

In a random-design analysis, the covariates \(X_1,\ldots,X_n\) are random, typically iid from a distribution \(G\), and the conditional response model is the logistic model given \(X_i\). The conditional information given the observed covariates is still the sum already derived. Its normalized limit is now a random-design statement: the realized design averages approach their population averages.

Proof. Each entry of the normalized conditional information is the sample average of the corresponding entry of \(w_\beta(X_i)(1,X_i)^\mathsf{T}(1,X_i)\). The entries are iid with finite absolute expectation, so the strong law of large numbers gives almost-sure convergence of each sample average to its expectation. There are only four entries, and the intersection of their four probability-one convergence events still has probability one. This proves the matrix convergence. For the expectation statement, linearity of expectation gives \(E[I_n]/n=n^{-1}\sum_{i=1}^n E[w_\beta(X_i)(1,X_i)^\mathsf{T}(1,X_i)]=J_G(\beta)\), because the covariates have the same distribution. \(\square\)

The expectation statement concerns information averaged over repeated covariate samples; the almost-sure statement concerns the information conditional on the particular covariates observed. In a random-design model, the joint likelihood also includes the covariate distribution. If that distribution does not involve \(\beta\), its log-likelihood contributes no information about \(\beta\), so the conditional information formula remains the relevant one.

What the Limit Tells Us

A fixed-design limit is determined by the limiting empirical distribution \(H\); a random-design limit is determined by the covariate distribution \(G\). If a sequence of fixed designs has empirical distributions approaching the same distribution as the random covariates, both information limits have the same integral form. The distinction is then in the sampling interpretation: fixed covariates have no sampling variation, whereas random covariates produce a realized information matrix that varies from sample to sample before converging.

For the logistic model, \(J\) is positive definite precisely when no nonzero vector \(a=(a_0,a_1)^\mathsf{T}\) satisfies \(a_0+a_1X=0\) with probability one under the limiting design distribution. Indeed, for any \(a\), \(a^\mathsf{T}J a=E[w_\beta(X)(a_0+a_1X)^2]\). At every finite predictor, \(w_\beta(X)>0\), so this expectation is zero exactly when the linear expression vanishes almost surely. Thus, with an intercept and slope, covariate values must vary in the limiting design to identify both directions from local information.

Under the usual likelihood regularity conditions and a nonsingular information limit, total information grows approximately as \(nJ\); consequently, the information-based covariance scale for the coefficient estimator is \(J^{-1}/n\). This gives a practical interpretation of the matrices calculated here: their eigen-directions describe how strongly the design informs combinations of coefficients. A small eigenvalue signals a weakly informed direction, even when the sample size is large.

Common Mistake: Averaging the Covariate First

A tempting but incorrect shortcut is to replace the covariate by its mean and calculate information only at that value. In general, \(E[w_\beta(X)z(X)z(X)^\mathsf{T}]\) is not equal to \(w_\beta(E[X])z(E[X])z(E[X])^\mathsf{T}\). The information weight depends nonlinearly on the covariate, and the matrix also contains \(X^2\). For example, in the continuous example, \(E[X]=0\) would make the slope component appear to vanish if one evaluated only at the mean; the correct component is \(E[X^2]/4=1/24\), which is positive.

The reliable calculation is to form the weighted matrix at each covariate value and then average its entries under the fixed-design limit or random-design distribution. This preserves both covariate variation and the response-dependent logistic weights.