Finite-Sample Calibration of Quasi-Likelihood Inference

Why Calibration Needs Attention

Quasi-likelihood methods use a specified mean and variance relationship to estimate regression effects and their uncertainty. Their familiar Wald tests and confidence intervals, however, are generally justified by large-sample approximations. When the sample is small, a few observations have high leverage, or responses are sparse, those approximations can be inaccurate even when the mean and variance assumptions are reasonable.

The key distinction is between an estimated variance and a calibrated reference distribution. A sandwich covariance estimate, a Pearson estimate of dispersion, or a leverage adjustment can improve an uncertainty estimate. None automatically makes a Wald statistic follow a normal, Student, or chi-square distribution in a finite sample.

What the Large-Sample Test Assumes

For independent observations, suppose the mean is $\mu_i(\beta)$ and the variance is $\phi V(\mu_i)$, where $\beta$ is a fixed-dimensional parameter vector and $\phi$ is a dispersion parameter. A quasi-score estimator solves an estimating equation built from the mean and variance specifications. Under suitable regularity conditions, it is approximately multivariate normal for large $n$. Robust Wald inference uses a consistent estimate of its covariance, including a sandwich covariance when the working variance may be wrong.

Asymptotic calibration: Let $\hat\beta$ estimate a parameter vector $\beta$, and let $\widehat{\operatorname{Var}}(\hat\beta)$ consistently estimate its covariance. For testing $H_0:C\beta=d$, with $C$ of rank $q$, the Wald statistic $$ W=(C\hat\beta-d)^\mathsf{T} \left\{C\widehat{\operatorname{Var}}(\hat\beta)C^\mathsf{T}\right\}^{-1} (C\hat\beta-d) $$ has an approximate $\chi^2_q$ distribution under $H_0$ as $n$ grows. This requires, among other conditions, a correctly targeted estimating equation, a nonsingular limiting covariance, an appropriate central limit theorem, and consistent covariance estimation. It is not a finite-sample identity.

In practice, the approximation is more vulnerable when the effective information is concentrated in only a few observations. Warning signs include low residual degrees of freedom, a large or unstable dispersion estimate, high leverage, rare events, many zero counts, or fitted means near a boundary. These signs do not prove that a test is wrong; they indicate that its reference distribution deserves scrutiny.

Leverage and Residual Variability

Leverage measures how strongly a design point can affect its own fitted value. Its connection to residuals explains why a large raw residual is not the only diagnostic for an influential observation: a high-leverage observation can have a small residual simply because the fitted value is drawn toward it.

Derived result: In a weighted linear approximation with fixed design matrix $X$ of rank $p$, suppose $\operatorname{Var}(Y)=\phi W^{-1}$, with $W$ positive definite. Define $P=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}$. The whitened residual vector $e=(I-P)W^{1/2}(Y-X\beta)$ has covariance $\phi(I-P)$. Consequently, its $i$th component has variance $\phi(1-h_{ii})$, where $h_{ii}=P_{ii}$.

Proof. Write $\varepsilon=Y-X\beta$. Then $e=(I-P)W^{1/2}\varepsilon$, and $\operatorname{Var}(W^{1/2}\varepsilon)=\phi I$. The matrix $P$ is symmetric and idempotent, so $$ \operatorname{Var}(e) =\phi(I-P)(I-P)^\mathsf{T} =\phi(I-P)^2 =\phi(I-P). $$ Taking diagonal entries gives $\operatorname{Var}(e_i)=\phi(1-P_{ii})$, as claimed. $\square$

For a generalized linear or quasi-likelihood fit, the weighted linear model is typically a local approximation, not an exact description of the response distribution. Its leverage values and residual adjustments are therefore useful diagnostics, not a general finite-sample theorem.

Worked Example: A High-Leverage Design Point

Consider an unweighted intercept-and-slope model with four predictor values $0,1,2,10$. The mean predictor is $\bar x=13/4=3.25$, and $$ S_{xx}=\sum_i(x_i-\bar x)^2 =10.5625+5.0625+1.5625+45.5625=62.75. $$

For the point $x=10$, its leverage is $$ h_{ii}=\frac{1}{4}+\frac{(10-3.25)^2}{62.75} =0.25+\frac{45.5625}{62.75} \approx 0.9761. $$

In the linear approximation, the residual variance at that point is only $\phi(1-h_{ii})\approx0.0239\phi$. Its residual standard deviation is therefore about $\sqrt{0.0239}=0.155$ times the ordinary error standard deviation. Dividing a residual by $\sqrt{1-h_{ii}}$, as in an HC2-style adjustment, multiplies it by about $6.47$. This flags how strongly the fitted value can absorb the point’s own response; it does not guarantee an exact test after adjustment.

Exact Calibration as a Benchmark

A Student $t$ reference distribution is exactly justified in a particular model, not in quasi-likelihood generally. The benchmark is useful because it shows which extra assumptions make a finite-sample correction exact.

Theorem: Suppose $Y=X\beta+\varepsilon$, where $X$ is fixed with rank $p$, $n>p$, and the errors are independent $N(0,\phi)$. For a nonzero contrast vector $c$, define $s^2=\operatorname{RSS}/(n-p)$, where $\operatorname{RSS}$ is the residual sum of squares. Under $H_0:c^\mathsf{T}\beta=c_0$, $$ T=\frac{c^\mathsf{T}\hat\beta-c_0} {s\sqrt{c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}} $$ has a Student $t_{n-p}$ distribution.

Proof. Under the stated normal model, the standardized contrast $$ Z=\frac{c^\mathsf{T}\hat\beta-c^\mathsf{T}\beta} {\sqrt{\phi\,c^\mathsf{T}(X^\mathsf{T}X)^{-1}c}} $$ is $N(0,1)$. The residual vector is the projection of $\varepsilon$ onto the orthogonal complement of the column space of $X$. That space has dimension $n-p$; hence $\operatorname{RSS}/\phi$ is $\chi^2_{n-p}$. The fitted and residual projections are orthogonal. Because they are jointly normal and have zero covariance, they are independent. Under the null, $T=Z/\sqrt{(\operatorname{RSS}/\phi)/(n-p)}$, the defining representation of a $t_{n-p}$ variable. $\square$

Quasi-likelihood specifies a mean and variance, not necessarily a normal response distribution or an exact chi-square law for a residual sum of squares. Replacing a normal critical value by a $t$ critical value can be a cautious approximation when dispersion is estimated, but it is not made exact by estimating $\phi$.

Worked Example: How Much a Small-Sample Reference Can Change an Interval

Suppose an estimated contrast is $0.80$, its estimated standard error is $0.35$, and the residual degrees of freedom are $9$. As an exact benchmark, assume the normal linear model conditions above hold. The two-sided 95% $t_9$ critical value is approximately $2.262$, so the interval is $$ 0.80\pm2.262(0.35)=0.80\pm0.7917, $$ or approximately $[0.008,1.592]$.

Using the large-sample normal critical value $1.96$ instead gives $$ 0.80\pm1.96(0.35)=0.80\pm0.686 =[0.114,1.486]. $$ The intervals differ appreciably, and the normal interval excludes more uncertainty. In a quasi-likelihood model, the $t_9$ calculation is not automatically exact; it illustrates the potential impact of finite-sample calibration, not a universal rule.

Cautious Calibration Strategies

There is no single correction appropriate for every quasi-likelihood fit. The choice should match the source of uncertainty and be reported as an approximation unless its assumptions supply an exact reference distribution.

Use degrees-of-freedom adjustments cautiously. A $t$ or $F$ reference can reflect estimated dispersion and limited residual degrees of freedom. Exactness requires stronger assumptions, such as the normal linear model conditions above. In other quasi-likelihood models, state that the reference is approximate.
Inspect leverage-adjusted covariance estimates. HC2- or HC3-style corrections increase the contribution of residuals at high-leverage observations. They can reduce a known small-sample bias in some settings, but do not ensure correct tail probabilities, particularly with very few observations or extreme leverage.
Consider a bootstrap only after specifying what is resampled. A parametric bootstrap requires a full response distribution, not just a quasi-variance function. A nonparametric bootstrap requires an appropriate sampling structure and can perform poorly for sparse data or small samples. Neither method removes the need to check whether its assumptions are credible.
Report sensitivity rather than a falsely precise answer. Compare reasonable covariance estimates or reference distributions, identify influential observations, and explain when conclusions depend on a calibration choice. For sparse responses, emphasize the limited information and consider whether the fitted mean model is supported by the observed data.

Worked Example: Sparse Counts and an Unspecified Response Law

Five independent count observations are $0,0,0,0,8$. Fit an intercept-only quasi-Poisson mean model, with $\mu_i=\mu$ and $\operatorname{Var}(Y_i)=\phi\mu_i$. The intercept score is proportional to $\sum_i(Y_i-\mu)$, so the fitted mean is $\hat\mu=8/5=1.6$, and the fitted log mean is $\hat\beta=\log(1.6)\approx0.470$.

The Pearson statistic is $$ X^2=\sum_i\frac{(Y_i-\hat\mu)^2}{\hat\mu} =4\frac{(0-1.6)^2}{1.6}+\frac{(8-1.6)^2}{1.6} =4(1.6)+25.6=32. $$ With $5-1=4$ residual degrees of freedom, the Pearson dispersion estimate is $\hat\phi=32/4=8$. The usual model-based standard error for the log mean is $\sqrt{\hat\phi/\sum_i\hat\mu_i}=\sqrt{8/8}=1$. The resulting Wald statistic for a zero log mean is only about $0.470$, but this calculation rests on a very sparse sample and an estimated dispersion dominated by the single count of 8.

The quasi-Poisson specification alone does not determine the probability of observing this pattern, so it cannot provide an exact small-sample reference distribution for the statistic. A Poisson parametric bootstrap would add a Poisson distributional assumption and would not reproduce arbitrary overdispersion. The defensible conclusion is to treat the Wald result cautiously, report the sparsity and dispersion estimate, and make any stronger calibration conditional on an explicitly stated response model.

Common Mistake: Treating a Correction as a Guarantee

A tempting argument is: “The sample is small, so I will use a $t$ distribution,” or “There is high leverage, so HC3 makes the test reliable.” The correction is that changing a covariance estimate or reference distribution does not by itself establish its finite-sample distribution. Exact $t$ calibration needs the assumptions of the stated theorem; leverage adjustments and small-sample $t$ approximations need to be described as approximations when those assumptions do not hold.

Calibration matters because a test’s reported significance level depends on its reference distribution, not just on whether its standard error looks plausible. When the data are sparse or information is concentrated, a careful account of assumptions and sensitivity is more informative than an unqualified $p$-value.

Check Your Understanding

Use these questions to check whether you can distinguish an uncertainty estimate from its calibration.

What conditions are needed for a robust Wald statistic for $q$ contrasts to have an asymptotic $\chi^2_q$ reference distribution?
Why can a high-leverage observation have a small raw residual, and what does an HC2-style adjustment change?
Under what assumptions is the Student $t$ contrast statistic in this tutorial exactly distributed as $t_{n-p}$?
Why does a quasi-variance specification alone not justify a parametric bootstrap for sparse counts?
Give two warning signs that should prompt a sensitivity analysis of quasi-likelihood inference.