Sandwich Covariance Under Model Misspecification

When a Likelihood Model Is Only an Approximation

A likelihood estimator is often calculated by maximizing a model that is not exactly the data-generating distribution. The resulting estimator can still converge to a well-defined parameter, but its variability need not be described by the inverse information matrix from the fitted model. The correct first-order covariance has a sandwich form: sensitivity to changes in the parameter on the outside, and variability of the score in the middle.

The distinction is consequential. If the fitted model is wrong, its curvature and the actual variation of its score are generally different quantities. The sandwich covariance keeps these two features separate instead of assuming the information identity that holds under correct specification.

Definition: Let $X_1,\ldots,X_n$ be independent and identically distributed from a true distribution $P_0$. Suppose a fitted parametric model has density or mass function $q_\theta(x)$, with parameter $\theta$ in an open subset of $\mathbb{R}^p$. The pseudo-true parameter $\theta^*$ is the unique maximizer, when it exists, of the expected log-likelihood $$ M(\theta)=E_0[\log q_\theta(X)], $$ where $E_0$ denotes expectation under $P_0$. When the relevant densities exist and the divergence is finite, this is also the parameter in the model minimizing Kullback–Leibler divergence from $P_0$.

The pseudo-true parameter need not describe the true distribution. It identifies the best approximation within the fitted family according to expected log-likelihood. Write $s_\theta(x)=\partial \log q_\theta(x)/\partial\theta$ for the column score vector, and let $H_\theta(x)$ be its $p$-by-$p$ derivative matrix. At an interior, differentiable maximizer, $E_0[s_{\theta^*}(X)]=0$, subject to conditions that justify differentiating the expectation.

Definition: At $\theta^*$, define the sensitivity matrix $A$ and score-variability matrix $B$ by $$ A=-E_0[H_{\theta^*}(X)], \qquad B=E_0[s_{\theta^*}(X)s_{\theta^*}(X)^\mathsf{T}]. $$ If the score has mean zero, $B$ is also its covariance matrix. When $A$ is nonsingular, the sandwich covariance is $$ V=A^{-1}B A^{-\mathsf{T}}, $$ where $A^{-\mathsf{T}}=(A^{-1})^\mathsf{T}$. For a scalar parameter this reduces to $V=B/A^2$.

Why the Sandwich Appears

For an independent sample, the log-likelihood is $\ell_n(\theta)=\sum_{i=1}^n\log q_\theta(X_i)$. A consistent interior likelihood estimator $\hat{\theta}$ solves the sample score equation $\sum_{i=1}^n s_{\hat{\theta}}(X_i)=0$. Expanding this equation around $\theta^*$ explains both parts of the covariance.

Theorem: Suppose $\theta^*$ is an interior pseudo-true parameter and $E_0[s_{\theta^*}(X)]=0$. Assume $\hat{\theta}\xrightarrow{p}\theta^*$, the score is differentiable near $\theta^*$, and the sample average of its derivative admits a uniform law of large numbers there, with limit $E_0[H_{\theta^*}(X)]=-A$. Assume the Taylor remainder for the sample score is $o_p(\sqrt n\|\hat{\theta}-\theta^*\|)$, $A$ is nonsingular, and $E_0[\|s_{\theta^*}(X)\|^2]<\infty$. Then $$ \sqrt n(\hat{\theta}-\theta^*)\xrightarrow{d}N(0,A^{-1}B A^{-\mathsf{T}}). $$

Proof. Write $\bar s_n(\theta)=n^{-1}\sum_{i=1}^n s_\theta(X_i)$. The score equation gives $\bar s_n(\hat{\theta})=0$. A Taylor expansion around $\theta^*$, with the stated remainder condition, gives

0=\bar s_n(\theta^*)+ \left\{\frac{1}{n}\sum_{i=1}^n H_{\tilde{\theta}_i}(X_i)\right\}(\hat{\theta}-\theta^*)+ o_p(\|\hat{\theta}-\theta^*\|),

where the intermediate parameter values lie between $\hat{\theta}$ and $\theta^*$; the uniform convergence condition makes the derivative matrix converge in probability to $-A$. Rearranging the expansion, or applying the corresponding local linearization, yields

\sqrt n(\hat{\theta}-\theta^*)= A^{-1}\frac{1}{\sqrt n}\sum_{i=1}^n s_{\theta^*}(X_i)+o_p(1).

The scores are independent and identically distributed, have mean zero, and have finite covariance $B$. The multivariate central limit theorem therefore gives

\frac{1}{\sqrt n}\sum_{i=1}^n s_{\theta^*}(X_i)\xrightarrow{d}N(0,B).

Multiplication by $A^{-1}$ transforms this limiting covariance to $A^{-1}B(A^{-1})^\mathsf{T}$. The negligible remainder does not change the limiting distribution by Slutsky's theorem. This proves the result. $\square$

In a sample, the corresponding covariance estimate replaces $A$ and $B$ by empirical versions, often evaluated at $\hat{\theta}$:

\widehat{A}=-\frac{1}{n}\sum_{i=1}^n H_{\hat{\theta}}(X_i), \qquad \widehat{B}=\frac{1}{n}\sum_{i=1}^n s_{\hat{\theta}}(X_i)s_{\hat{\theta}}(X_i)^\mathsf{T}, \qquad \widehat{\operatorname{Cov}}(\hat{\theta})=\frac{1}{n}\widehat{A}^{-1}\widehat{B}\widehat{A}^{-\mathsf{T}}.

The factor $1/n$ matters: $V$ is the limiting covariance of $\sqrt n(\hat{\theta}-\theta^*)$, while the estimator itself has first-order covariance $V/n$.

Worked Example: A Normal Location Model With the Wrong Variance

Suppose the fitted model is $q_\mu=N(\mu,1)$, but the true distribution puts probability $1/2$ on each of $-2$ and $2$. The fitted log-likelihood, apart from a constant, is $-\frac12\sum_i(X_i-\mu)^2$, so its maximizer is $\hat{\mu}=\bar X$.

The expected log-likelihood is maximized at the true mean, which is zero: explicitly, $E_0[(X-\mu)^2]=4+\mu^2$, minimized at $\mu^*=0$. The score is $s_\mu(X)=X-\mu$, and its derivative is $-1$. Thus at $\mu^*=0$,

A=-E_0[-1]=1,\qquad B=E_0[X^2]=4,\qquad V=\frac{B}{A^2}=4.

Indeed, $\sqrt n\bar X$ converges to $N(0,4)$. The model-based inverse information would instead report variance $1$ for $\sqrt n\hat{\mu}$, because the fitted normal model has variance one. It understates the actual first-order variance by a factor of four.

Curvature Is Not Score Variability

Under correct specification, expected negative curvature and score variability agree under standard regularity conditions. Under misspecification, they are defined using different aspects of the true data distribution: $A$ describes how the expected score changes with the parameter, while $B$ describes how scores vary across observations. There is no general reason for them to be equal.

Result: If the model is correctly specified at $\theta_0$, the model density is twice differentiable, its support does not depend on $\theta$, and differentiation can be interchanged with integration, then $$ -E_{\theta_0}[H_{\theta_0}(X)] = E_{\theta_0}[s_{\theta_0}(X)s_{\theta_0}(X)^\mathsf{T}]. $$ This is the information identity.

Derivation. For each $x$, differentiating $q_\theta(x)$ gives $\partial q_\theta(x)/\partial\theta=q_\theta(x)s_\theta(x)$. Differentiating once more yields the matrix identity

\frac{\partial^2 q_\theta(x)}{\partial\theta\,\partial\theta^\mathsf{T}} =q_\theta(x)\{H_\theta(x)+s_\theta(x)s_\theta(x)^\mathsf{T}\}.

Because $q_\theta$ integrates to one for every $\theta$, its integrated second derivative is the zero matrix. The stated interchange condition allows that derivative to pass under the integral. Evaluating at the correctly specified $\theta_0$ gives

0=E_{\theta_0}[H_{\theta_0}(X)] +E_{\theta_0}[s_{\theta_0}(X)s_{\theta_0}(X)^\mathsf{T}],

which is the identity. With misspecification, the expectation is under $P_0$, not under $q_{\theta^*}$. The normalization $\int q_\theta(x)\,dx=1$ therefore does not imply that the corresponding expectation under $P_0$ is zero. $\square$

Worked Example: A Poisson Likelihood for Overdispersed Counts

Let the fitted model be Poisson with parameter $\lambda>0$, but suppose the true distribution has $P_0(X=0)=3/4$ and $P_0(X=4)=1/4$. The true mean is $E_0[X]=1$, while $E_0[X^2]=4$, so $\operatorname{Var}_0(X)=4-1=3$.

The Poisson log-likelihood has score $s_\lambda(X)=X/\lambda-1$, and its sample maximizer is $\hat{\lambda}=\bar X$. The expected log-likelihood is maximized at $\lambda^*=E_0[X]=1$: its derivative is $E_0[X]/\lambda-1$, positive for $\lambda<1$ and negative for $\lambda>1$. The negative expected score derivative and score second moment at that value are

A=E_0\left[\frac{X}{\lambda^2}\right]_{\lambda=1}=1, \qquad B=E_0[(X-1)^2]=3.

Hence $V=A^{-2}B=3$, which agrees with the variance of the sample mean under the true distribution. The inverse model-based information is $1/\lambda^*=1$. Using it would ignore the true count variation and understate the asymptotic variance by a factor of three.

Worked Example: Estimating Normal Mean and Variance From Uniform Data

Fit a normal model $N(\mu,v)$, with $v>0$, to independent data from the uniform distribution on $[-\sqrt{3},\sqrt{3}]$. The true mean is zero, the true variance is one, and the fourth moment is $E_0[X^4]=9/5$. The expected normal log-likelihood, omitting a constant, is

M(\mu,v)=-\frac12\log v-\frac{E_0[(X-\mu)^2]}{2v}.

For fixed $v$, $E_0[(X-\mu)^2]=1+\mu^2$ is minimized at $\mu=0$. With $\mu=0$, differentiating $M(0,v)=-\frac12\log v-\frac{1}{2v}$ gives $-1/(2v)+1/(2v^2)$, which vanishes at $v=1$ and changes from positive to negative there. Thus $(\mu^*,v^*)=(0,1)$.

For parameter order $(\mu,v)$, the score components are $s_\mu=(X-\mu)/v$ and $s_v=-1/(2v)+(X-\mu)^2/(2v^2)$. At $(0,1)$, symmetry gives zero cross terms in both matrices. The negative expected score derivative and score second moment are

A= \begin{pmatrix} 1&0\\ 0&1/2 \end{pmatrix}, \qquad B= \begin{pmatrix} E_0[X^2]&E_0[X(X^2-1)/2]\\ E_0[X(X^2-1)/2]&E_0[(X^2-1)^2/4] \end{pmatrix} = \begin{pmatrix} 1&0\\ 0&1/5 \end{pmatrix}.

In the last entry, $E_0[(X^2-1)^2]/4=(9/5-2+1)/4=1/5$. Therefore

A^{-1}BA^{-\mathsf{T}} = \begin{pmatrix} 1&0\\ 0&4/5 \end{pmatrix}.

The model-based inverse information is $A^{-1}=\operatorname{diag}(1,2)$, so it gives asymptotic variance $2$ for the variance estimator instead of the correct $4/5$. The discrepancy comes from the uniform distribution's fourth moment, which differs from the normal model's fourth moment.

Practical Interpretation and a Common Pitfall

The sandwich does not repair a misspecified model or make its pseudo-true parameter equal to a scientifically desired target. It describes first-order sampling variability around $\theta^*$, under the stated sampling and regularity conditions. Whether $\theta^*$ is substantively useful remains a separate modeling question.

Common mistake: reporting only the inverse Hessian because the estimator came from maximizing a likelihood. That reasoning silently assumes that the fitted likelihood is the true sampling distribution, or that the information identity holds. Under misspecification, the Hessian estimates $A$, not $B$; inverse curvature alone generally omits the actual score variability. The correction is to estimate both components and use $\widehat A^{-1}\widehat B\widehat A^{-\mathsf{T}}/n$. In a correctly specified regular model, $A=B$, and the sandwich reduces to the familiar inverse-information covariance.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

In words, what does the pseudo-true parameter optimize, and why need it not be a true parameter of the data-generating distribution?
For a scalar estimator, identify $A$ and $B$ and state the sandwich variance of $\sqrt n(\hat{\theta}-\theta^*)$.
Why does the Poisson example have asymptotic variance three even though its fitted model-based information is one?
State the key density normalization argument used to derive the information identity.
For the uniform-data example, what feature of the true distribution determines the sandwich variance of the estimated normal variance?

Sandwich Covariance Under Model Misspecification

What You'll Learn