Sensitivity, Variability, and Godambe Information

Why Sensitivity Is Not the Whole Uncertainty Calculation

A quasi-score uses a specified mean and variance relationship to form an estimating equation, without requiring a complete probability distribution for the observations. Its sensitivity describes how the expected estimating equation changes as the parameter changes. Its variability describes how much the estimating equation itself fluctuates from sample to sample. These are different features, and both are needed to assess the uncertainty associated with a solution of the equation.

When the working variance is correct, sensitivity and variability coincide for the quasi-score considered here. If the actual variance differs from the working variance, they generally do not. The resulting combination is called Godambe information; its inverse is the covariance matrix of the first-order linear approximation to the estimator. This tutorial derives these quantities, rather than developing the large-sample theory that can justify their use as an asymptotic covariance.

Quasi-Score, Sensitivity, and Variability

Let $Y_1,\ldots,Y_n$ be independent observations, and let $\beta$ be a $p$-dimensional parameter. Suppose the mean model is $E[Y_i]=\mu_i(\beta)$, where each $\mu_i$ is differentiable. The quasi-score uses a working variance $w_i(\beta)=\phi V(\mu_i(\beta))$, with $w_i(\beta)>0$ and fixed dispersion $\phi$. Write $D_i(\beta)=\partial\mu_i(\beta)/\partial\beta$, a $p$-dimensional column vector. The estimating function is

U(\beta)=\sum_{i=1}^{n}\frac{D_i(\beta)}{w_i(\beta)}\{Y_i-\mu_i(\beta)\}.

The estimate solves $U(\hat{\beta})=0$. For the calculations below, assume the mean model is correct at the parameter value $\beta_0$, so $E[Y_i]=\mu_i(\beta_0)$. Denote the actual variance at that value by $\sigma_i^2=\operatorname{Var}(Y_i)$. It need not equal the working variance $w_i(\beta_0)$.

Definition: The sensitivity matrix is $H=-E[\partial U(\beta_0)/\partial\beta^T]$. The variability matrix is $J=\operatorname{Var}(U(\beta_0))$. When $H$ and $J$ are nonsingular, the Godambe information for this estimating function is $G=H^T J^{-1}H$. For the quasi-score below, $H$ is symmetric, so $G=HJ^{-1}H$.

The minus sign in $H$ is a convention: a well-behaved estimating function typically decreases, in expectation, as the parameter moves away from its target. In this setting its expected derivative is negative $H$. The definition concerns the estimating function, not a claim that the quasi-score is the derivative of a full log-likelihood.

Deriving the Two Matrices

To differentiate a summand of $U$, both $D_i/w_i$ and the residual $Y_i-\mu_i$ may depend on $\beta$. The product rule therefore produces terms involving derivatives of $D_i/w_i$, as well as the derivative of the residual. At $\beta_0$, the first group of terms has expectation zero because $E[Y_i-\mu_i(\beta_0)]=0$. The remaining term gives the sensitivity.

Theorem: Suppose the observations are independent, the mean functions and working variances are differentiable near $\beta_0$, $w_i(\beta_0)>0$, and the expectations and derivatives in the sensitivity definition exist and may be interchanged. If $E[Y_i]=\mu_i(\beta_0)$, then $$ H=\sum_{i=1}^{n}\frac{D_iD_i^T}{w_i}, \qquad J=\sum_{i=1}^{n}\frac{\sigma_i^2}{w_i^2}D_iD_i^T, $$ where $D_i=D_i(\beta_0)$ and $w_i=w_i(\beta_0)$.

Proof. Set $a_i(\beta)=D_i(\beta)/w_i(\beta)$. The $i$th summand of $U$ is $a_i(\beta)\{Y_i-\mu_i(\beta)\}$. Differentiating with respect to $\beta^T$ gives

\frac{\partial}{\partial\beta^T} \left[a_i(\beta)\{Y_i-\mu_i(\beta)\}\right] = \frac{\partial a_i(\beta)}{\partial\beta^T}\{Y_i-\mu_i(\beta)\} -a_i(\beta)D_i(\beta)^T.

At $\beta_0$, the expectation of the first term is zero: its derivative factor is nonrandom, and the residual has expectation zero. The expectation of the second term is $-D_iD_i^T/w_i$. Summing and changing sign proves the formula for $H$. For $J$, each summand of $U(\beta_0)$ has mean zero. Independence makes the covariance of the sum the sum of the covariances, and

\operatorname{Var}\left(\frac{D_i}{w_i}\{Y_i-\mu_i\}\right) = \frac{D_iD_i^T}{w_i^2}\operatorname{Var}(Y_i) = \frac{\sigma_i^2}{w_i^2}D_iD_i^T.

Adding these terms proves the formula for $J$. $\square$

The derivation shows why the two matrices can differ. The sensitivity depends on the working variance in the denominator, whereas variability also depends on the actual variance of the observations. If the observations are not independent, $J$ must additionally include the cross-covariance terms between distinct summands.

From the Matrices to Godambe Information

If $H$ is nonsingular, a first-order expansion of the estimating equation around $\beta_0$ gives a useful covariance calculation. The expansion is an approximation, not by itself a large-sample result:

0=U(\hat{\beta}) \ \approx\ U(\beta_0)-H(\hat{\beta}-\beta_0), \qquad \hat{\beta}-\beta_0\ \approx\ H^{-1}U(\beta_0).

Consequently, the covariance matrix of this linear approximation is

\operatorname{Var}\{H^{-1}U(\beta_0)\} = H^{-1}J(H^{-1})^T.

For the quasi-score, $H$ is symmetric, so this becomes $H^{-1}JH^{-1}$. Its inverse is $HJ^{-1}H=G$, the Godambe information. If $H$ is not symmetric for a more general estimating function, the corresponding definitions use $G=H^TJ^{-1}H$ and covariance $H^{-1}J(H^{-1})^T$. Nonsingularity is required for these inverses; in applications, identifiability and the design must also provide enough information for $H$ to be nonsingular.

Key distinction: $H^{-1}$ is inverse sensitivity, not generally the covariance matrix. The covariance of the first-order approximation is $H^{-1}JH^{-1}$ for a symmetric $H$. These agree only when the variability and sensitivity matrices match.

Worked Calculations

Worked Example: A Scalar Mean With a Misspecified Working Variance

Suppose $Y_1,Y_2$ are independent with common mean $\beta$, but their actual variances are $1$ and $9$, respectively. Use working variance $w_i=4$ for each observation. Calculate $H$, $J$, the Godambe information, and the covariance of the first-order approximation.

Here $D_1=D_2=1$, so the estimating equation is $U(\beta)=\{Y_1-\beta+Y_2-\beta\}/4$. The sensitivity is

H=\frac{1}{4}+\frac{1}{4}=\frac{1}{2}.

The actual variances enter $J$, giving

J=\frac{1}{4^2}+\frac{9}{4^2} =\frac{10}{16} =\frac{5}{8}.

Therefore

G=H^2J^{-1} =\frac{1}{4}\cdot\frac{8}{5} =\frac{2}{5}, \qquad G^{-1}=\frac{5}{2}=2.5.

By contrast, inverse sensitivity is $H^{-1}=2$, not $2.5$. In fact, solving $U(\beta)=0$ gives $\hat{\beta}=(Y_1+Y_2)/2$, whose variance is $(1+9)/4=2.5$. This verifies the sandwich calculation directly and shows that the discrepancy is not just a matrix-notation issue.

Worked Example: A Two-Parameter Regression Equation

Consider two independent observations with mean model $\mu_i=x_i^T\beta$, where $x_1=(1,0)^T$, $x_2=(1,1)^T$, and $\beta=(\beta_0,\beta_1)^T$. Use working variances $w_1=w_2=1$, while the actual variances are $\sigma_1^2=1$ and $\sigma_2^2=4$. Find $H$, $J$, and the covariance matrix of the first-order approximation.

Since $D_i=x_i$, the sensitivity is

H=x_1x_1^T+x_2x_2^T = \begin{pmatrix}1&0\\0&0\end{pmatrix} + \begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}2&1\\1&1\end{pmatrix}.

The variability weights each outer product by the actual variance:

J=x_1x_1^T+4x_2x_2^T = \begin{pmatrix}1&0\\0&0\end{pmatrix} + \begin{pmatrix}4&4\\4&4\end{pmatrix} = \begin{pmatrix}5&4\\4&4\end{pmatrix}.

The determinant of $H$ is $2(1)-1(1)=1$, so

H^{-1}= \begin{pmatrix}1&-1\\-1&2\end{pmatrix}.

Multiplying out the sandwich gives

H^{-1}JH^{-1} = \begin{pmatrix}1&-1\\-1&2\end{pmatrix} \begin{pmatrix}5&4\\4&4\end{pmatrix} \begin{pmatrix}1&-1\\-1&2\end{pmatrix} = \begin{pmatrix}1&-1\\-1&5\end{pmatrix}.

For a check, the middle product of the first two matrices is $\begin{pmatrix}1&0\\3&4\end{pmatrix}$, which yields the displayed symmetric result. Inverse sensitivity alone would instead give $\begin{pmatrix}1&-1\\-1&2\end{pmatrix}$. The Godambe information is the inverse of the sandwich covariance:

G= \begin{pmatrix}1&-1\\-1&5\end{pmatrix}^{-1} = \frac{1}{4}\begin{pmatrix}5&1\\1&1\end{pmatrix}.

Worked Example: When Sensitivity and Variability Match

Let $Y_1,Y_2$ be independent Poisson observations with means $\mu_i=\exp(\beta x_i)$, where $x_1=1$, $x_2=2$, and evaluate the matrices at $\beta_0=0$. Use the Poisson variance function $w_i=\mu_i$ with $\phi=1$.

At $\beta_0=0$, both means are $1$, and $D_i=\partial\mu_i/\partial\beta=\mu_i x_i$, giving $D_1=1$ and $D_2=2$. Thus

H=\frac{D_1^2}{w_1}+\frac{D_2^2}{w_2} =\frac{1}{1}+\frac{4}{1}=5.

The actual Poisson variances equal the working variances, so

J=\frac{1\cdot D_1^2}{w_1^2} +\frac{1\cdot D_2^2}{w_2^2} =1+4=5.

Therefore $G=H^2/J=5$, and the covariance of the first-order approximation is $G^{-1}=1/5$. Here $H^{-1}=1/5$ happens to agree with the covariance because $J=H$, not because inverse sensitivity is always a covariance.

When Does Inverse Sensitivity Work?

For this quasi-score, if the working variance is the true variance for every observation, then $\sigma_i^2=w_i$. The derived formulas immediately imply

J=\sum_{i=1}^{n}\frac{D_iD_i^T}{w_i}=H.

In that case $G=H$, and the first-order covariance is $H^{-1}$. If the actual variance differs from the working one, this identity generally fails. The formula for $J$ still uses the actual variance, while $H$ continues to be determined by the working variance. A correct mean model alone does not force $J=H$.

A common mistake is to identify $H^{-1}$ as the estimator covariance just because $H$ is called information or resembles a likelihood information matrix. That reasoning silently assumes $J=H$. The correction is to calculate the variability of the estimating function from the actual observation variances, then use $H^{-1}JH^{-1}$ for the first-order covariance calculation. In practice, $J$ may need to be estimated from data when the actual variances are unknown; the derivations here identify what quantity must be estimated, not a particular estimation procedure.

These matrices matter because they separate two questions: how responsive the estimating equation is to parameter changes, and how noisy that equation is under repeated sampling. Godambe information combines both. It therefore provides the relevant information measure for an estimating equation even when a full likelihood is unavailable, while keeping the assumptions behind a covariance calculation visible.

Check Your Understanding

Use the definitions and derivations above to answer the following.

For independent observations, explain why the derivative of $D_i/w_i$ contributes no term to the expected sensitivity when the mean model is correct.
What changes in the formula for $J$ if the actual variance of an observation is twice its working variance?
State the first-order covariance formula for a quasi-score with symmetric nonsingular sensitivity $H$, and identify the condition under which it reduces to $H^{-1}$.
Why can $H^{-1}$ not generally be interpreted as the covariance merely because $H$ measures sensitivity?
If observations are dependent, what additional contributions must be considered when calculating $J$?