Common Errors in GLM Asymptotic Arguments

Why Asymptotic Normality Needs Conditions

A normal approximation for a GLM estimator is a conclusion, not an automatic consequence of having a large number of observations. Its familiar form depends on a regular, identifiable parameter, an estimator that solves the score equation near the true parameter, and information that grows at the appropriate rate. Misspecification can change the covariance, while a boundary or weakly identified parameter can invalidate the usual argument altogether.

The useful habit is to check the argument’s separate ingredients rather than to ask only whether $n$ is large. The examples below show what fails in several common cases and how to distinguish a failed regularity condition from a valid but approximate large-sample result.

The Regular Score Argument in Brief

Let $Y_1,\ldots,Y_n$ be independent observations, and let $\theta\in\mathbb{R}^p$ be the parameter of interest. Write $q_i(\theta)$ for the contribution of observation $i$ to the log-likelihood or estimating objective, and define its score contribution as $s_i(\theta)=\partial q_i(\theta)/\partial\theta$. The average score is $S_n(\theta)=n^{-1}\sum_{i=1}^n s_i(\theta)$. A maximum-likelihood estimator in the interior of the parameter space satisfies $S_n(\hat\theta)=0$.

Regular asymptotic normality conditions: A standard score-based argument requires, at a minimum, a well-defined identifiable target $\theta^\ast$ in the interior of the parameter space; consistency of $\hat\theta$; differentiability of the score near $\theta^\ast$; a score central limit theorem $\sqrt n S_n(\theta^\ast)\xrightarrow{d}N(0,K)$; and convergence of the average score derivative along the segment between $\theta^\ast$ and $\hat\theta$ to $-J$, where $J$ is nonsingular. For independent, identically distributed observations under suitable moment and regularity conditions, $K=\operatorname{Var}(s_i(\theta^\ast))$ and $J=-E[\partial s_i(\theta^\ast)/\partial\theta^T]$. The matrices $K$ and $J$ need not be equal.

Under these conditions, the first-order expansion produces the sandwich covariance. The proof is short, but each assumption has a job; the failure examples that follow correspond to losing one of those jobs.

Result: If $S_n(\hat\theta)=0$, $\hat\theta\xrightarrow{p}\theta^\ast$, $\sqrt n S_n(\theta^\ast)\xrightarrow{d}N(0,K)$, and the average score derivative on the intervening segment converges in probability to a nonsingular matrix $-J$, then $$ \sqrt n(\hat\theta-\theta^\ast)\xrightarrow{d}N\!\left(0,J^{-1}KJ^{-1}\right), $$ where $J$ is symmetric. If the score equation is only approximately solved, the same conclusion requires its residual to be negligible at the $n^{-1/2}$ scale.

Proof. By the fundamental theorem of calculus, define $A_n=\int_0^1 \frac{\partial S_n(\theta^\ast+t(\hat\theta-\theta^\ast))}{\partial\theta^T}\,dt$ as the average score derivative along the segment from $\theta^\ast$ to $\hat\theta$. Then

0=S_n(\hat\theta)=S_n(\theta^\ast)+A_n(\hat\theta-\theta^\ast), \qquad A_n\xrightarrow{p}-J.

Since $J$ is nonsingular, $A_n$ is invertible with probability tending to one, and $A_n^{-1}\xrightarrow{p}-J^{-1}$. Rearranging and multiplying by $\sqrt n$ gives

\sqrt n(\hat\theta-\theta^\ast)=-A_n^{-1}\sqrt n S_n(\theta^\ast).

The score central limit theorem and Slutsky’s theorem therefore give a limiting normal distribution with covariance $(-J^{-1})K(-J^{-1})^T=J^{-1}KJ^{-1}$, since $J$ is symmetric. $\square$

Worked Diagnostics

Worked Example: Non-Identifiable Poisson Coefficients

Suppose $Y_1,\ldots,Y_n$ are independent Poisson observations with $\log E[Y_i]=\beta_1+\beta_2$ for every $i$. The design gives the two coefficients identical columns. Determine whether the usual two-parameter asymptotic covariance can be obtained.

Write $\lambda=\exp(\beta_1+\beta_2)$. The likelihood is $$ L(\beta_1,\beta_2)=\prod_{i=1}^n\frac{\exp(-\lambda)\lambda^{Y_i}}{Y_i!}. $$ It depends on $\beta_1,\beta_2$ only through their sum. For any real $c$, the pair $(\beta_1+c,\beta_2-c)$ gives exactly the same $\lambda$, and therefore exactly the same distribution of the data. The two coefficients are not separately identifiable.

The expected information for $(\beta_1,\beta_2)$ is $$ I_n(\beta_1,\beta_2)=n\lambda \begin{pmatrix}1&1\\1&1\end{pmatrix}. $$ Its determinant is zero, since its two rows are identical. Thus it has no inverse, and an information-based covariance for the two separate coefficients is undefined. More observations multiply the matrix by $n$ but do not remove its rank deficiency. One can reparameterize in terms of the identifiable sum $\beta_1+\beta_2$, or change the design; one cannot justify separate coefficient normal limits by simply increasing $n$.

Worked Example: An Estimate at the Boundary

Let $Y_1,\ldots,Y_n$ be independent Bernoulli observations with success probability $p\in[0,1]$, and suppose the true value is $p=0$. Find the maximum-likelihood estimate and explain why the usual interior normal argument does not apply.

At $p=0$, $Y_i=0$ with probability one for every $i$. Consequently, the observed sample consists entirely of zeros with probability one. For a sample of all zeros, the likelihood is $L(p)=(1-p)^n$, which is maximized over $[0,1]$ at $\hat p=0$. Hence $\hat p=0$ almost surely under the true distribution, for every $n$.

The usual score expansion assumes an interior target and a locally differentiable score equation that the estimator solves. Here the target lies at the edge of the parameter space, and the maximum is a constrained boundary maximum. The ordinary interior formula involving $1/[p(1-p)]$ is not even finite at $p=0$. The exact degeneracy of this example is not evidence for a regular normal approximation with a standard error obtained by substituting $p=0$; it is evidence that the interior regularity conditions fail.

Worked Example: Information That Does Not Grow

Consider independent Poisson observations with $\log E[Y_i]=\beta x_i$, where $x_i=1/i$, and suppose the true value is $\beta=0$. Does increasing $n$ provide the usual growing information for estimating $\beta$?

At $\beta=0$, each $Y_i$ has a Poisson distribution with mean 1. The score contribution is $x_i(Y_i-\exp(\beta x_i))$, and the expected information from observation $i$ at the true value is $x_i^2\exp(0)=1/i^2$. Thus the total expected information from the first $n$ observations is $$ I_n(0)=\sum_{i=1}^n\frac{1}{i^2}\leq \sum_{i=1}^{\infty}\frac{1}{i^2}=\frac{\pi^2}{6}. $$

Although the sample size tends to infinity, the information remains bounded rather than diverging. The observations increasingly have covariates close to zero, so they provide little information about the slope. The standard argument that normalizes estimation error by the square root of information cannot be replaced by a routine $\sqrt n$ argument here: $\sqrt{I_n(0)}$ does not grow without bound. A valid asymptotic analysis would need a different design or a different asymptotic regime.

Misspecification: Curvature and Variability Can Differ

When the likelihood model is wrong, a well-defined interior pseudo-true parameter may still have a normal estimator. The score’s variability and the objective’s curvature must then be kept separate. Let $\theta^\ast$ be an interior maximizer of the expected log-likelihood, with $E[s_i(\theta^\ast)]=0$. Under suitable differentiability, moment, consistency, and central limit conditions, define $K=\operatorname{Var}(s_i(\theta^\ast))$ and $J=-E[\partial s_i(\theta^\ast)/\partial\theta^T]$. The score central limit theorem has covariance $K$, while the derivative converges to $-J$. The resulting covariance is $J^{-1}KJ^{-1}$, not automatically $J^{-1}$.

Worked Example: A Poisson Likelihood for Bernoulli Data

Suppose $Y_i$ are independent Bernoulli variables with success probability $1/2$, but fit a Poisson intercept model with mean $\lambda=\exp(\beta)$. Find the pseudo-true value and compare the model-based and sandwich asymptotic variances for $\hat\beta$.

The fitted model’s score contribution is $s_i(\beta)=Y_i-\exp(\beta)$. Its expectation under the actual Bernoulli distribution is $1/2-\exp(\beta)$, which vanishes at $\beta^\ast=\log(1/2)$. At this value, $$ J=-E\!\left[\frac{\partial s_i(\beta^\ast)}{\partial\beta}\right] =E[\exp(\beta^\ast)]=\frac12, \qquad K=\operatorname{Var}(s_i(\beta^\ast)) =\operatorname{Var}(Y_i)=\frac14. $$

Therefore the sandwich variance for $\sqrt n(\hat\beta-\beta^\ast)$ is $$ J^{-1}KJ^{-1}=2\left(\frac14\right)2=1. $$ The model-based inverse-information variance is $J^{-1}=2$, which differs. This also agrees with a direct check: when the sample mean is positive, the Poisson intercept estimate is $\hat\beta=\log(\bar Y)$; the Bernoulli sample mean has asymptotic variance $1/4$ on the $\sqrt n$ scale, and the derivative of $\log(x)$ at $x=1/2$ is 2, giving variance $2^2(1/4)=1$. The event that all observations are zero has probability $2^{-n}$; it tends to zero and does not change this limiting calculation if the estimator is defined arbitrarily there. The example shows why a score CLT must use covariance $K$, separately from the derivative limit $-J$.

When Does Information Equality Hold?

Under correct specification, expected curvature and score variability coincide under regularity conditions. This is not a general identity that can be carried over unchanged to a misspecified likelihood.

Result: Suppose $Y$ has density or mass function $f_\theta(y)$ with respect to a common dominating measure, the model is correctly specified at an interior $\theta$, and $f_\theta$ is twice differentiable in a neighborhood of $\theta$. If differentiation can be interchanged with integration (or summation) twice, the relevant derivatives are integrable, and the score has finite second moments, then $$ E_\theta[s(Y;\theta)]=0,\qquad -E_\theta\!\left[\frac{\partial s(Y;\theta)}{\partial\theta^T}\right] =E_\theta[s(Y;\theta)s(Y;\theta)^T]. $$

Proof. Since the density integrates or sums to one, differentiation under the integral gives $\int \partial f_\theta(y)/\partial\theta\,dy=0$. Because $s(y;\theta)=\partial\log f_\theta(y)/\partial\theta =(\partial f_\theta(y)/\partial\theta)/f_\theta(y)$ wherever $f_\theta(y)>0$, this is $E_\theta[s(Y;\theta)]=0$. For the second identity, differentiate the log density twice: $$ \frac{\partial^2\log f_\theta(y)}{\partial\theta\,\partial\theta^T} =\frac{1}{f_\theta(y)} \frac{\partial^2 f_\theta(y)}{\partial\theta\,\partial\theta^T} -s(y;\theta)s(y;\theta)^T. $$ Taking expectations, the first term integrates to $\int \partial^2 f_\theta(y)/(\partial\theta\,\partial\theta^T)\,dy=0$, again by the stated interchange condition and the fact that the density integrates to one. Thus the expected Hessian of the log density is $-E_\theta[s(Y;\theta)s(Y;\theta)^T]$. Multiplying by $-1$ proves the identity. $\square$

Under misspecification, the fitted density is not the data-generating density, so the normalization argument in this proof does not establish equality between $J$ and $K$. The score may have mean zero at the pseudo-true value, but that fact alone does not equate its variance with the expected negative derivative.

A Limit Is Not an Exact Finite-Sample Law

Worked Example: A Discrete Statistic and a Continuous Limit

Let $Y_1,\ldots,Y_4$ be independent Bernoulli observations with $p=1/2$. What is the exact distribution of $\sqrt{4}(\bar Y-p)$, and why is it not exactly normal?

The count $4\bar Y$ has a Binomial$(4,1/2)$ distribution. Therefore $\bar Y$ takes values $0,1/4,1/2,3/4,1$ with respective probabilities $1/16,4/16,6/16,4/16,1/16$. The statistic $2(\bar Y-1/2)$ consequently takes values $-1,-1/2,0,1/2,1$ with those same probabilities. In particular, its probability of equaling zero is $6/16$.

A continuous normal distribution assigns probability zero to any single point, including zero. The exact finite-sample law is thus not the normal distribution. The central limit theorem says that $\sqrt n(\bar Y-p)$ converges in distribution to $N(0,p(1-p))$ as $n$ grows; it does not say that the distribution at $n=4$, or at any finite $n$, is exactly normal.

Common Mistakes and a Practical Check

A frequent error is to state “the MLE is asymptotically normal, so its covariance is the inverse expected information” without checking identifiability, the parameter-space boundary, information growth, or the relationship between $J$ and $K$. Another is to treat a limiting approximation as an exact finite-sample distribution. The correction is to identify which assumptions establish the limit, and which additional conditions justify the particular covariance formula being used.

Check identification and location.
Confirm the parameter is identifiable and the target is interior. If the information matrix is singular or the estimate is constrained to a boundary, do not use the standard interior covariance formula without a separate argument.

Check information growth.
Verify that the information in the asymptotic regime grows sufficiently for the proposed normalization and a concentrating estimator. A large count of weak observations need not imply large information.

Separate score variability from curvature.
Write the score limit with covariance $K$ and the derivative limit as $-J$. Use $J^{-1}$ only when the required information equality holds; otherwise use the sandwich covariance $J^{-1}KJ^{-1}$, subject to its regularity conditions.

Keep the finite-sample qualification.
Describe a normal distribution as a limiting or approximate distribution unless an exact finite-sample result has separately been proved.

Key takeaway: Asymptotic normality is conditional. Check identification, interiority, information growth, and the separate limits for score variability and curvature before using a normal approximation or an information-based covariance.

Check Your Understanding

For each question, state which part of the asymptotic argument is at issue.

A Poisson model has two identical covariate columns. Why does increasing the sample size not identify the two corresponding coefficients separately?
In the Bernoulli boundary example with true $p=0$, what is the maximum-likelihood estimate, and which regularity condition fails?
For a Poisson log-mean model with $x_i=1/i$, calculate the total expected information at $\beta=0$ and explain its behavior as $n$ increases.
Under misspecification, what do $J$ and $K$ represent, and what covariance follows from the score expansion?
Why does a central limit theorem not imply that a statistic has an exact normal distribution at a fixed finite sample size?