Large-Sample Theory for Quasi-Likelihood Estimators

Why Large-Sample Conditions Matter

A quasi-likelihood estimator is obtained by solving an estimating equation, not necessarily by maximizing a probability likelihood. Its large-sample behavior therefore depends on the behavior of the estimating function and its derivative. In particular, consistency needs both identification and a root that actually satisfies the equation with high probability. Asymptotic normality additionally requires a central limit theorem and control of the derivative between the true parameter and the estimated root.

Let \(Y_{n1},\ldots,Y_{nn}\) be independent observations in row \(n\), and let \(\theta\in\mathbb{R}^p\) be the parameter. The row index permits fixed or changing designs. Write \(\psi_{ni}(\theta)\in\mathbb{R}^p\) for observation \(i\)'s estimating-function contribution, and define

\Psi_n(\theta)=\frac{1}{n}\sum_{i=1}^n\psi_{ni}(\theta).

A quasi-score is a particular estimating function. For example, with independent responses, mean \(\mu_i(\theta)\), derivative row vector \(D_i(\theta)=\partial\mu_i(\theta)/\partial\theta^T\), and working variance \(V_i(\theta)\), a common contribution is \(D_i(\theta)^T V_i(\theta)^{-1}\{Y_i-\mu_i(\theta)\}\). The results below apply to this and other estimating functions; they do not require a fully specified distribution for each response.

Consistency: Identification and Root Existence

Definition: An estimator \(\hat\theta_n\) is an estimating-equation root with probability tending to one if \(P\{\Psi_n(\hat\theta_n)=0\}\to 1\). It is consistent for \(\theta_0\) if \(\hat\theta_n\xrightarrow{p}\theta_0\). These are separate properties: an estimator need not be a root on every outcome, but the probability of not being a root must tend to zero for the root-based arguments below.

Here is a useful consistency result. Let \(K\subset\mathbb{R}^p\) be compact with \(\theta_0\) in its interior. Suppose there is a continuous function \(g:K\to\mathbb{R}^p\) such that

\sup_{\theta\in K}\|\Psi_n(\theta)-g(\theta)\|\xrightarrow{p}0, \qquad g(\theta_0)=0,

and \(\theta_0\) is the only zero of \(g\) in \(K\). Suppose also that \(P\{\hat\theta_n\in K,\ \Psi_n(\hat\theta_n)=0\}\to1\). Then \(\hat\theta_n\xrightarrow{p}\theta_0\).

The uniform convergence condition is often established by a uniform law of large numbers for independent observations. In applications, suitable integrable envelope and continuity or Lipschitz conditions can provide such a law. The theorem states the convergence it needs directly, rather than assuming that independence alone is enough.

Result (Consistency of roots): Under the conditions just stated, every estimator that lies in \(K\) and solves the estimating equation with probability tending to one is consistent for \(\theta_0\).

Proof. Fix \(\varepsilon>0\). If \(K_\varepsilon=\{\theta\in K:\|\theta-\theta_0\|\geq\varepsilon\}\) is nonempty, compactness and continuity imply that \(\|g(\theta)\|\) attains a minimum on \(K_\varepsilon\). Since \(g\) has no zero there, this minimum is some \(\eta_\varepsilon>0\). On the event \(\sup_{\theta\in K}\|\Psi_n(\theta)-g(\theta)\|<\eta_\varepsilon/2\), every \(\theta\in K_\varepsilon\) satisfies

\|\Psi_n(\theta)\| \geq \|g(\theta)\|-\|\Psi_n(\theta)-g(\theta)\| > \eta_\varepsilon/2.

Thus no such \(\theta\) can be a root on that event. Its probability tends to one by uniform convergence. The event that \(\hat\theta_n\in K\) and is a root also has probability tending to one, so \(P(\|\hat\theta_n-\theta_0\|\geq\varepsilon)\to0\). If \(K_\varepsilon\) is empty, that probability is already bounded by \(P(\hat\theta_n\notin K)\to0\). This holds for every \(\varepsilon>0\), proving consistency. \(\square\)

Worked Example: A Bernoulli Quasi-Score Root

Let \(Y_1,\ldots,Y_n\) be independent Bernoulli observations with success probability \(p_0\), where \(0<p_0<1\). Use the mean \(\mu(\theta)=\theta\) and Bernoulli variance function \(V(\theta)=\theta(1-\theta)\). For \(0<\theta<1\), the quasi-score contribution is \(\psi_i(\theta)=(Y_i-\theta)/\{\theta(1-\theta)\}\).

For any \(\theta\in(0,1)\), the denominator is positive, so \(\Psi_n(\theta)=0\) exactly when \(\sum_i(Y_i-\theta)=0\). The root is therefore \(\hat p_n=\bar Y\), provided \(0<\bar Y<1\). The probability that this interior root does not exist is \(P(\bar Y=0)+P(\bar Y=1)=(1-p_0)^n+p_0^n\to0\). Thus the root-existence condition holds, and the law of large numbers gives \(\bar Y\xrightarrow{p}p_0\).

The population estimating function is \(g(\theta)=(p_0-\theta)/\{\theta(1-\theta)\}\), whose only zero in any compact subinterval of \((0,1)\) containing \(p_0\) is \(p_0\). The example shows why it is appropriate to require a root with probability tending to one, rather than incorrectly requiring an interior root on every possible sample.

Asymptotic Normality and the Sandwich Limit

For the normal limit, define the sensitivity matrix \(A\) as the negative limiting average derivative of the estimating function, and \(B\) as the limiting average covariance of its contributions. Use \(\dot\psi_{ni}(\theta)\) for the \(p\times p\) derivative matrix. The following conditions make the derivative control strong enough to apply even when consistency has been established but a root-\(n\) rate has not yet been shown.

Theorem (Asymptotic normality of estimating-equation roots): Suppose \(\hat\theta_n\xrightarrow{p}\theta_0\) and \(P\{\Psi_n(\hat\theta_n)=0\}\to1\). Assume:

Each \(\psi_{ni}\) is continuously differentiable on a fixed open ball around \(\theta_0\), and \(E[\psi_{ni}(\theta_0)]=0\).
For some nonsingular \(p\times p\) matrix \(A\) and some \(\delta>0\), \(\sup_{\|\theta-\theta_0\|\leq\delta}\|n^{-1}\sum_i\dot\psi_{ni}(\theta)+A\|\xrightarrow{p}0\).
The independent contributions satisfy \(n^{-1}\sum_i\operatorname{Cov}\{\psi_{ni}(\theta_0)\}\to B\), and \(n^{-1}\sum_iE\|\psi_{ni}(\theta_0)\|^{2+\gamma}\) is bounded for some \(\gamma>0\).

Then \(\sqrt n(\hat\theta_n-\theta_0)\xrightarrow{d}N(0,A^{-1}BA^{-T})\).

The bounded \(2+\gamma\) moment condition is a convenient sufficient condition for the Lindeberg condition; it can be replaced by that condition directly. The limit matrix \(B\) may be singular, in which case the normal limit can be degenerate. In one dimension the sandwich variance is \(B/A^2\).

Proof. By independence, the zero-mean condition, covariance convergence, and the stated moment bound, the multivariate Lindeberg–Feller central limit theorem gives

\frac{1}{\sqrt n}\sum_{i=1}^n\psi_{ni}(\theta_0) \xrightarrow{d}N(0,B).

For clarity, the moment bound implies the Lindeberg condition: for any \(c>0\),

\frac1n\sum_i E\!\left[ \|\psi_{ni}(\theta_0)\|^2 \mathbf{1}\{\|\psi_{ni}(\theta_0)\|>c\sqrt n\} \right] \leq \frac{1}{c^\gamma n^{\gamma/2}}\, \frac1n\sum_i E\|\psi_{ni}(\theta_0)\|^{2+\gamma} \longrightarrow0.

On the event that the equation holds and \(\hat\theta_n\) lies in the \(\delta\)-ball, the fundamental theorem of calculus along the line segment from \(\theta_0\) to \(\hat\theta_n\) yields

0=\Psi_n(\theta_0)+ \left\{\int_0^1 \frac1n\sum_{i=1}^n \dot\psi_{ni}\bigl(\theta_0+t(\hat\theta_n-\theta_0)\bigr)\,dt \right\}(\hat\theta_n-\theta_0).

Consistency puts the whole segment in that fixed ball with probability tending to one. The uniform derivative condition therefore makes the matrix in braces converge in probability to \(-A\). Since \(A\) is nonsingular, that matrix is invertible with probability tending to one, and its inverse converges in probability to \(-A^{-1}\). Rearranging on the event where the equation holds and the inverse exists gives

\sqrt n(\hat\theta_n-\theta_0) = \left\{-\int_0^1\frac1n\sum_i \dot\psi_{ni}\bigl(\theta_0+t(\hat\theta_n-\theta_0)\bigr)\,dt\right\}^{-1} \frac1{\sqrt n}\sum_i\psi_{ni}(\theta_0).

The matrix factor converges in probability to \(A^{-1}\); the equation and invertibility events both have probability tending to one. Slutsky's theorem now gives the stated limit, whose covariance is \(A^{-1}B(A^{-1})^T=A^{-1}BA^{-T}\). \(\square\)

Three Sandwich Calculations

Worked Example: A Weighted Mean With Correct Working Variances

For even \(n\), suppose the \(Y_i\) are independent with \(E[Y_i]=\theta_0\). Half have variance \(1\), and half have variance \(4\). Use the weighted estimating function \(\psi_i(\theta)=(Y_i-\theta)/v_i\), where \(v_i\) is the corresponding variance, either \(1\) or \(4\).

The derivative is \(\dot\psi_i(\theta)=-1/v_i\), so \(A=n^{-1}\sum_i1/v_i=(1+1/4)/2=5/8\). The variance of \(\psi_i(\theta_0)\) is \(\operatorname{Var}(Y_i)/v_i^2=1/v_i\), giving \(B=5/8\). Hence the limiting variance of \(\sqrt n(\hat\theta_n-\theta_0)\) is \(A^{-1}BA^{-1}=8/5\). Directly, the root is \(\hat\theta_n=\sum_iY_i/v_i\,/\,\sum_i1/v_i\); its variance is \(1/\sum_i1/v_i=8/(5n)\), which confirms the sandwich calculation.

Worked Example: An Unweighted Mean Under Unequal Variances

Keep the same independent observations and true variances as in the preceding calculation, but use \(\psi_i(\theta)=Y_i-\theta\), which gives every observation equal working weight. Then \(\dot\psi_i=-1\), so \(A=1\). The average variability is \(B=n^{-1}\sum_i\operatorname{Var}(Y_i)=(1+4)/2=5/2\).

The root is \(\bar Y\), and the theorem gives \(\sqrt n(\bar Y-\theta_0)\xrightarrow{d}N(0,5/2)\), so its variance is approximately \(5/(2n)\). In general, \(A\) describes how quickly the mean estimating function changes with \(\theta\), whereas \(B\) describes the actual variability of that function. Replacing \(B\) by \(A\) would incorrectly report variance \(1/n\) in this example.

Worked Example: A Linear Mean Estimating Equation

Consider the scalar regression mean \(E[Y_i]=x_i\theta_0\) with independent errors of mean zero. For even \(n\), let half the covariates be \(x_i=1\) with error variance \(1\), and half be \(x_i=2\) with error variance \(4\). Use the unweighted estimating function \(\psi_i(\theta)=x_i(Y_i-x_i\theta)\).

Its derivative is \(-x_i^2\), so \(A=n^{-1}\sum_i x_i^2=(1+4)/2=5/2\). The contribution's variance is \(x_i^2\operatorname{Var}(Y_i)\): it equals \(1\) in the first group and \(2^2\cdot4=16\) in the second. Thus \(B=(1+16)/2=17/2\), and the sandwich variance is \((17/2)/(5/2)^2=34/25=1.36\).

Indeed, the root is \(\hat\theta_n=\sum_i x_iY_i/\sum_i x_i^2\). Its variance is \(\sum_i x_i^2\operatorname{Var}(Y_i)/(\sum_i x_i^2)^2 =n(17/2)/\{n(5/2)\}^2=34/(25n)\), matching the asymptotic calculation.

Common Pitfalls and Interpretation

The sandwich form separates two features of the estimating equation: sensitivity \(A\), which determines how a change in the parameter affects the equation, and variability \(B\), which measures the randomness of the equation at the true parameter. They coincide in some correctly specified likelihood settings, but they need not coincide for a general quasi-score or working variance. That distinction is why a robust covariance calculation uses both.

Using only a shrinking-neighborhood derivative condition. A derivative approximation established only when \(\|\theta-\theta_0\|\leq C/\sqrt n\) cannot be applied along the segment to an estimator known merely to be consistent. Consistency alone does not establish a root-\(n\) rate. The theorem instead requires uniform derivative convergence on a fixed neighborhood.
Assuming a root exists whenever an estimator is defined. A chosen fallback value on samples with no root does not make that value a root. The consistency and normality results require the probability that the estimator is a root to tend to one (or a separate argument controlling the exceptional event).
Confusing a working variance with actual variability. Working variances can determine the estimating weights, but the covariance \(B\) is the variability of the resulting estimating-function contributions under the actual sampling process.

Key takeaway: Consistency follows from uniform convergence, unique identification, and a root that exists with probability tending to one. The sandwich normal limit additionally requires a central limit theorem and derivative control on a fixed neighborhood of the true parameter.

Check Your Understanding

Use the definitions and conditions above to answer the following.

Why does uniform separation of \(g(\theta)\) from zero outside an \(\varepsilon\)-neighborhood help prove consistency?
What is the difference between saying an estimator is defined and saying it solves its estimating equation with probability tending to one?
In the asymptotic-normality theorem, why is uniform derivative convergence on a fixed neighborhood useful when consistency is known but a root-\(n\) rate is not?
For the unweighted mean example, identify the values of \(A\) and \(B\), and explain why they differ.
What additional sampling condition makes the moment bound in the theorem useful for a central limit theorem?