Sandwich Variance from the Asymptotic Linear Representation

Why Two Matrices Determine the Limit

An estimator defined by a vector estimating equation is affected by two different sources of variation. The first is how sharply the population equation changes when the parameter moves; the second is the random fluctuation of the equation across observations. These quantities need not coincide, and their separate roles produce the sandwich covariance.

This tutorial uses the convention that the sensitivity matrix is the negative expected derivative of the estimating function. With this convention, the asymptotic linear representation has a positive leading term, and the covariance takes the familiar form $A^{-1}BA^{-T}$. The matrix $A^{-T}$ means $(A^{-1})^T$.

Definition: Let $W_1,\ldots,W_n$ be independent, identically distributed observations, and let $\psi(W,\theta)$ be a $p$-dimensional estimating function for a $p$-dimensional parameter $\theta$. Suppose the target $\theta_0$ satisfies $E[\psi(W,\theta_0)]=0$. The sensitivity matrix and score-variability matrix are $$ A=-E\!\left[\frac{\partial\psi(W,\theta_0)}{\partial\theta^T}\right], \qquad B=E[\psi(W,\theta_0)\psi(W,\theta_0)^T]. $$ When $E[\psi(W,\theta_0)]=0$, $B$ is also the covariance matrix of $\psi(W,\theta_0)$.

The empirical equation is $n^{-1}\sum_{i=1}^n\psi(W_i,\hat\theta)=0$. The derivative in the definition of $A$ is a $p$-by-$p$ matrix, with rows corresponding to components of $\psi$ and columns corresponding to components of $\theta$. The matrix $B$ describes the variation in one observation’s estimating function, not the variation in the derivative.

From the Estimating Equation to the Sandwich

The key step is to linearize the empirical equation around $\theta_0$. The result below makes the regularity requirements explicit. It is the asymptotic linear representation used to obtain the covariance.

Theorem: Suppose $W_1,\ldots,W_n$ are iid, $E[\psi(W,\theta_0)]=0$, and $E[\|\psi(W,\theta_0)\|^2]<\infty$. Suppose $\hat\theta\xrightarrow{p}\theta_0$, the empirical estimating equation has residual $o_p(n^{-1/2})$, and $\psi(W,\theta)$ is differentiable in a neighborhood of $\theta_0$. Assume the empirical Jacobian converges uniformly in that neighborhood, as it shrinks to $\theta_0$, to the population Jacobian at $\theta_0$. If $A$ exists and is nonsingular, then $$ \sqrt n(\hat\theta-\theta_0) = A^{-1}\frac{1}{\sqrt n}\sum_{i=1}^n\psi(W_i,\theta_0)+o_p(1). $$ If $B$ is finite, then $$ \sqrt n(\hat\theta-\theta_0)\xrightarrow{d} N\!\left(0,A^{-1}BA^{-T}\right). $$

Here the empirical Jacobian condition means that, for parameter values between $\theta_0$ and $\hat\theta$, the matrix $n^{-1}\sum_i\partial\psi(W_i,\theta)/\partial\theta^T$ converges in probability to $E[\partial\psi(W,\theta_0)/\partial\theta^T]=-A$. This condition, together with consistency, controls the Taylor expansion at the random estimator.

Proof. Write $\Psi_n(\theta)=n^{-1}\sum_{i=1}^n\psi(W_i,\theta)$. By the multivariate integral form of Taylor’s theorem,

\Psi_n(\hat\theta) = \Psi_n(\theta_0)+ \left\{\int_0^1 \frac{\partial\Psi_n(\theta_0+t(\hat\theta-\theta_0))} {\partial\theta^T}\,dt\right\}(\hat\theta-\theta_0).

Call the matrix in braces $J_n$. The empirical Jacobian condition and consistency imply $J_n\xrightarrow{p}-A$. The estimating equation residual is $o_p(n^{-1/2})$, so rearranging gives

\sqrt n(\hat\theta-\theta_0) = (-J_n)^{-1}\sqrt n\,\Psi_n(\theta_0)+o_p(1).

Because $A$ is nonsingular and $J_n\xrightarrow{p}-A$, the inverse exists with probability tending to one and $(-J_n)^{-1}\xrightarrow{p}A^{-1}$. The multivariate central limit theorem applies to the iid, mean-zero vectors $\psi(W_i,\theta_0)$ with finite covariance $B$; hence $\sqrt n\,\Psi_n(\theta_0)\xrightarrow{d}N(0,B)$. Slutsky’s theorem now gives both the stated linear representation and its limiting normal distribution. $\square$

Interpretation: The estimating function fluctuates at order $n^{-1/2}$, with covariance $B$ for its scaled sum. The inverse sensitivity $A^{-1}$ translates that fluctuation into parameter movement. Consequently, the covariance is $A^{-1}BA^{-T}$, with one inverse sensitivity on each side of the variability matrix.

Worked Examples

Worked Example: The Sample Mean

Let $W_i$ be iid with mean $\mu$ and finite variance $\sigma^2$. Define $\psi(W,\theta)=W-\theta$. The estimating equation is $n^{-1}\sum_i(W_i-\hat\theta)=0$, whose solution is $\hat\theta=\bar W$.

At $\theta_0=\mu$, the estimating function has mean zero. Its derivative with respect to $\theta$ is $-1$, so $A=-E[-1]=1$. Its variability is $B=E[(W-\mu)^2]=\sigma^2$. Thus

A^{-1}BA^{-T}=1^{-1}\sigma^2 1^{-1}=\sigma^2, \qquad \sqrt n(\bar W-\mu)\xrightarrow{d}N(0,\sigma^2).

The variance of $\bar W$ is therefore approximately $\sigma^2/n$, while the covariance in the limit for the scaled estimator is $\sigma^2$. This distinction between the two scales is essential.

Worked Example: A Transformed Two-Component Moment Equation

Let $W=(W_1,W_2)^T\sim N((\theta_{01},\theta_{02})^T,\operatorname{diag}(1,4))$, with independent components. Consider $\psi(W,\theta)=C(W-\theta)$, where

C= \begin{pmatrix} 1&1\\ 0&2 \end{pmatrix}.

Since $C$ is nonsingular, the empirical equation has the same root as the componentwise sample-mean equation. The derivative is $-C$, so $A=C$. At $\theta_0$, the covariance of $W-\theta_0$ is $\Sigma=\operatorname{diag}(1,4)$, and therefore

B=C\Sigma C^T = \begin{pmatrix}1&4\\0&8\end{pmatrix} \begin{pmatrix}1&0\\1&2\end{pmatrix} = \begin{pmatrix}5&8\\8&16\end{pmatrix}.

The inverse sensitivity and sandwich are

A^{-1}= \begin{pmatrix}1&-\tfrac12\\0&\tfrac12\end{pmatrix}, \qquad A^{-1}BA^{-T} = \begin{pmatrix}1&-\tfrac12\\0&\tfrac12\end{pmatrix} \begin{pmatrix}5&8\\8&16\end{pmatrix} \begin{pmatrix}1&0\\-\tfrac12&\tfrac12\end{pmatrix} = \begin{pmatrix}1&0\\0&4\end{pmatrix}.

The transformed estimating function has a non-diagonal variability matrix, but the inverse sensitivity transforms it back to the covariance of the sample mean. This example also shows why the order of the matrices matters.

Worked Example: Heteroskedastic Linear Regression

Let $Z$ equal $1$ or $-1$, each with probability $1/2$, and define $X=(1,Z)^T$. Suppose $Y=X^T\beta_0+\varepsilon$, where $E[\varepsilon\mid Z]=0$, $\operatorname{Var}(\varepsilon\mid Z=1)=1$, and $\operatorname{Var}(\varepsilon\mid Z=-1)=4$. Use the estimating function $\psi((Y,Z),\beta)=X(Y-X^T\beta)$.

At $\beta_0$, $\psi=X\varepsilon$ has mean zero. Its derivative is $-XX^T$, giving

A=E[XX^T] = \frac12 \begin{pmatrix}1&1\\1&1\end{pmatrix} + \frac12 \begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}1&0\\0&1\end{pmatrix}.

For $B=E[XX^T\varepsilon^2]$, the conditional mean-zero assumption ensures that the conditional second moment of $\varepsilon$ equals its conditional variance. Thus

B=\frac12(1) \begin{pmatrix}1&1\\1&1\end{pmatrix} +\frac12(4) \begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}2.5&-1.5\\-1.5&2.5\end{pmatrix}.

Since $A=I$, the sandwich covariance is $B$. Its eigenvalues are $2.5-1.5=1$ and $2.5+1.5=4$, both positive. The scaled coefficient estimator therefore has limiting covariance $B$, and its covariance is approximately $B/n$. Replacing the unequal conditional variances by a single common variance would miss both the off-diagonal term and the difference in coefficient uncertainty.

Useful Properties and a Common Pitfall

The sandwich is a valid covariance matrix whenever $B$ is a covariance matrix and $A$ is nonsingular. For any fixed contrast vector $c$, its asymptotic variance follows directly from the matrix expression.

Result: If $B$ is positive semidefinite and $A$ is nonsingular, then $V=A^{-1}BA^{-T}$ is positive semidefinite. For any vector $c$, the asymptotic variance of $c^T\sqrt n(\hat\theta-\theta_0)$ is $c^TVc$.

Proof. Set $u=A^{-T}c$. Then

c^TVc =c^TA^{-1}BA^{-T}c =u^TBu\geq 0,

because $B$ is positive semidefinite. The contrast statement follows by applying the linear transformation $c^T$ to the limiting normal vector. $\square$

A common mistake is to write $A^{-1}BA^{-1}$ without checking matrix order. In a vector problem, $A$ need not be symmetric, so the right-hand factor must be $A^{-T}$, not $A^{-1}$. Another mistake is to treat $B$ as the derivative matrix: $B$ measures the variability of the estimating function, whereas $A$ measures its expected slope. They are distinct except under additional structure. The sandwich gives the covariance of the scaled estimator; for the unscaled estimator, the corresponding first-order covariance is $A^{-1}BA^{-T}/n$.

Key takeaway: First calculate the negative expected derivative $A$, then the covariance $B$ of one observation’s estimating function. The asymptotic covariance is $A^{-1}BA^{-T}$, with sensitivity on both sides of score variability.

Check Your Understanding

Use the definitions and derivation above to answer these questions.

For a vector estimating equation, what does $A$ measure, and what does $B$ measure?
Why does the sandwich expression use $A^{-T}$ on the right?
If $A=I$ and $B$ is a covariance matrix, what is the asymptotic covariance of the scaled estimator?
In the regression example, why does $E[\varepsilon\mid Z]=0$ matter when calculating $B$?
How does the approximate covariance of $\hat\theta$ differ from the limiting covariance of $\sqrt n(\hat\theta-\theta_0)$?

Sandwich Variance from the Asymptotic Linear Representation

What You'll Learn