Worked Asymptotics for Bernoulli Logistic Regression

Why Work Through Logistic Regression Asymptotics?

The general large-sample theory becomes concrete when applied to a particular likelihood. In Bernoulli logistic regression, the score and information have especially direct forms: each observation contributes its covariate vector multiplied by a residual, and its information contribution is a weighted outer product. These formulas show exactly how the fitted probabilities and the design determine coefficient uncertainty.

The goal here is to derive those formulas and connect them to the asymptotic distribution of the maximum-likelihood estimator. We consider independent responses with a fixed-dimensional parameter and fixed covariates. The asymptotic result requires regularity conditions; having a logistic link alone is not enough.

Model, Likelihood, and Information

For an observed response \(y_i\in\{0,1\}\), the Bernoulli probability mass is \(\pi_i(\beta)^{y_i}\{1-\pi_i(\beta)\}^{1-y_i}\). Independence gives the log-likelihood

Differentiating each summand uses \(\partial\pi_i(\beta)/\partial\beta=\pi_i(\beta)\{1-\pi_i(\beta)\}x_i\). The resulting score and observed information are

At \(\beta_0\), \(E[Y_i]=\pi_i(\beta_0)\), so \(E[U_n(\beta_0)]=0\). Independence and \(\operatorname{Var}(Y_i)=\pi_i(\beta_0)\{1-\pi_i(\beta_0)\}\) give

Thus, under correct specification, expected and observed information have the same formula in this model: the observed information does not depend on the responses. This identity does not make the observed information nonrandom in general; here it is determined by the covariates and the parameter at which it is evaluated.

Asymptotic Normality Under Explicit Conditions

A large-sample covariance statement needs conditions that control both the design and the likelihood. The following fixed-design result is a useful worked version of the general likelihood argument. Assume throughout that \(p\) is fixed, the \(x_i\) are deterministic, and \(\sup_i\|x_i\|<\infty\). Also assume the model is correctly specified at a finite \(\beta_0\), and that

Suppose the maximum-likelihood estimator \(\hat\beta\) is finite with probability tending to one and is consistent for \(\beta_0\). These assumptions exclude asymptotic degeneracy of the information and ensure that the likelihood score vanishes at the fitted estimator with probability tending to one. Bounded covariates and finite \(\beta_0\) keep the true probabilities away from 0 and 1; the positive-definite limit requires information in every parameter direction.

Proof. First, the score at the true parameter is a sum of independent mean-zero vectors \(x_i\{Y_i-\pi_i(\beta_0)\}\), with covariance \(J_n(\beta_0)\). For any fixed vector \(a\in\mathbb R^p\), the scalar summands in \(a^TU_n(\beta_0)\) are uniformly bounded, because the \(x_i\) are bounded and \(|Y_i-\pi_i(\beta_0)|\leq1\). Their total variance divided by \(n\) tends to \(a^TJa\). When \(a\ne0\), this limit is positive. The Lindeberg condition therefore holds: for each \(\epsilon>0\), all summands are smaller than \(\epsilon\sqrt n\) for sufficiently large \(n\), so the sum of their truncated second moments is zero. The scalar central limit theorem and the Cramér–Wold device yield

Next use the integral Taylor formula for the vector-valued score. With probability tending to one, \(U_n(\hat\beta)=0\), and

This formula integrates the Hessian along the line segment; it does not presume that a single intermediate point represents the entire matrix. Denote the bracketed matrix by \(A_n\). To show \(A_n/n\xrightarrow{p}J\), note that the derivative of \(w(\eta)=\operatorname{logit}^{-1}(\eta)\{1-\operatorname{logit}^{-1}(\eta)\}\) is bounded on the real line. The mean-value bound and bounded \(\|x_i\|\) imply, for a constant \(C\) independent of \(i,n\) and \(\beta\),

Consistency makes the right side tend to zero in probability uniformly for \(\beta=\beta_0+t(\hat\beta-\beta_0)\), \(0\leq t\leq1\). Hence \(A_n/n\xrightarrow{p}J\). Since \(J\) is positive definite, matrix inversion is continuous at \(J\), and \((A_n/n)^{-1}\xrightarrow{p}J^{-1}\). Rearranging the integral score equation gives

by Slutsky’s theorem. Finally, the same consistency and bounded-derivative argument gives \(J_n(\hat\beta)/n\xrightarrow{p}J\), so \(J_n(\hat\beta)^{-1}\) estimates the leading covariance \(J^{-1}/n\). \(\square\)

Uncertainty for a Linear Contrast

For a fixed vector \(c\in\mathbb R^p\), the contrast \(c^T\beta\) is a scalar coefficient combination. Applying the linear transformation \(c^T\) to the asymptotic normal limit gives

The corresponding estimated standard error is

This is a covariance calculation: the off-diagonal entries of the coefficient covariance matrix matter whenever the contrast combines coefficients. Adding coefficient variances while ignoring their covariance is generally incorrect.

Common Pitfall and Practical Interpretation

A common mistake is to claim that the vector score’s Taylor expansion has one common intermediate point \(\tilde\beta\) at which the whole Hessian equals the integrated change in score. The scalar mean-value theorem does not generally supply such a point for a vector-valued function. The correction is to use the integral Taylor formula, as in the proof: it gives the exact averaged-information matrix along the segment from \(\beta_0\) to \(\hat\beta\). Consistency and the stated boundedness conditions then show that this matrix, divided by \(n\), converges to \(J\).

Another important distinction is between the covariance approximation and the coefficient estimate itself. \(J_n(\hat\beta)^{-1}\) estimates the covariance of \(\hat\beta\), while \(J_n(\hat\beta)/n\) estimates the limiting information per observation. Under the stated conditions, information grows proportionally to \(n\), so coefficient standard errors shrink at the rate \(n^{-1/2}\). This conclusion relies on information growing in every parameter direction; a nearly singular information matrix signals weakly determined coefficient combinations.