Why Two Standard Errors Can Accompany One Poisson Fit
A Poisson frequency model can give a useful estimate of claim frequency even when observed counts vary more than a Poisson distribution predicts. In that situation, the fitted coefficients need not change, but their estimated uncertainty can. The distinction matters: a coefficient standard error describes uncertainty in the estimated coefficient, not the fitted mean itself.
This tutorial compares the conventional, or model-based, standard error with the HC0 sandwich standard error for Poisson regression. HC0 is a heteroskedasticity-consistent estimator that allows the conditional variance of each independent observation to differ from its Poisson-model value. Its validity still depends on a correctly specified conditional mean and independent observations. It does not correct a wrongly specified mean model or dependence between observations.
The exposure appears as an offset in the mean, not as an additional estimated coefficient. For a single observation, the Poisson score contribution for \(\beta\) is \(x_i(Y_i-\mu_i)\). The model-based calculation uses the Poisson variance \(\operatorname{Var}(Y_i\mid x_i)=\mu_i\). HC0 instead uses the squared fitted residual \((Y_i-\widehat{\mu}_i)^2\) to estimate the variability in that score contribution.
Deriving the Sandwich Form
The sandwich formula follows from a first-order approximation to the estimating equation. Let \(U(\beta)=\sum_i x_i(Y_i-\mu_i)\) be the score. At the fitted value, \(U(\widehat{\beta})=0\). A Taylor expansion around the true coefficient \(\beta\) gives, approximately,
Taking the variance of the approximation gives \(A^{-1}\operatorname{Var}(U)A^{-1}\). With independent observations, the variance of the summed score is the sum of the score variances. If the conditional mean is correct but the conditional variances need not equal \(\mu_i\), then the score variance is estimated by the residual-based matrix \(B\). Substitution gives \(A^{-1}BA^{-1}\).
If the Poisson distribution is correctly specified, each score contribution has variance \(\mu_i x_i x_i^\mathsf{T}\), so the expected score-variance matrix equals \(A\). Substituting \(B=A\) into the sandwich expression reduces it to \(A^{-1}\), the model-based covariance. This explains why the two estimators target the same asymptotic covariance under a correctly specified Poisson model, while they can differ when the variance assumption fails.
Worked Example: Intercept-Only Rate With Moderate Residual Variation
Worked Example: Intercept-Only Rate With Moderate Residual Variation
A hypothetical insurer observes four independent, unit-exposure policies with claim counts \(1,3,5,7\). Fit an intercept-only Poisson model and calculate the estimated rate, the model-based standard error of the log rate, and the HC0 standard error. Compare the results.
Method and fitted rate. In an intercept-only model, every policy has fitted mean \(\widehat{\mu}=\exp(\widehat{\beta})\). The Poisson maximum-likelihood estimate is total count divided by total exposure:
Model-based standard error. The one-dimensional information, or bread, is the sum of the fitted means: \(A=4+4+4+4=16\). Thus the estimated variance of the log rate is \(A^{-1}=1/16=0.0625\), and its standard error is \(\sqrt{0.0625}=0.2500\).
HC0 standard error. The fitted residuals are \(1-4=-3\), \(3-4=-1\), \(5-4=1\), and \(7-4=3\). Their squared sum is \(9+1+1+9=20\), so \(B=20\). The sandwich variance is \(A^{-1}BA^{-1}=20/16^2=20/256=0.078125\). Its square root is \(0.279508\ldots\), or 0.2795 to four decimals.
The fitted rate is 4 under either calculation. The estimated log-rate standard error is 0.2500 under the Poisson variance assumption and 0.2795 under HC0. The latter is larger because the observed squared residuals sum to more than the Poisson-based bread, 20 compared with 16.
Check and common mistake. The model-based variance is the reciprocal of the total fitted mean, \(1/16\), not \(1/4\). Taking the square root of \(1/16\) gives 0.2500. Using \(1/4\) as the variance would confuse the fitted rate with the total information and produce an incorrect standard error.
Worked Example: The Residual Pattern Can Matter
Worked Example: The Residual Pattern Can Matter
For a second hypothetical portfolio, four independent unit-exposure policies have counts \(0,2,6,8\). Calculate both standard errors for the fitted log rate and identify whether the HC0 estimate indicates more or less coefficient uncertainty than the model-based estimate.
Fit the model. The fitted rate is \((0+2+6+8)/4=16/4=4\), so each fitted mean is 4 and \(\widehat{\beta}=\log(4)\). The bread is \(A=\sum_i\widehat{\mu}_i=4+4+4+4=16\). The model-based variance is \(1/16=0.0625\), giving a standard error of \(\sqrt{0.0625}=\mathbf{0.2500}\).
Calculate HC0. The residuals are \(0-4=-4\), \(2-4=-2\), \(6-4=2\), and \(8-4=4\). The squared residuals sum to \(16+4+4+16=40\), so \(B=40\). The sandwich variance is \(40/16^2=40/256=0.15625\), and the standard error is \(\sqrt{0.15625}=0.395284\ldots\), or 0.3953 to four decimals.
Here HC0 gives a larger standard error than the model-based calculation. Both fit the same rate of 4; the difference arises because the observed residual variation is greater than the Poisson variance used in the conventional calculation.
Check and common mistake. The bread remains 16, so the model-based standard error is \(\sqrt{1/16}=0.2500\). For HC0, the residual-square sum is 40, not 16, giving variance \(40/256\). Treating the HC0 meat as though it were the bread would incorrectly make both standard errors 0.2500.
Worked Example: A Binary Rating Factor
Worked Example: A Binary Rating Factor
A hypothetical book has eight independent unit-exposure policies. Four policies are in rating group 0 and have counts \(0,2,4,6\); four are in group 1 and have counts \(2,4,6,8\). Fit a Poisson model with an intercept and a group indicator. Calculate model-based and HC0 standard errors for the intercept and group coefficient.
Fit the coefficients. The fitted mean in group 0 is its average count, \((0+2+4+6)/4=12/4=3\). The fitted mean in group 1 is \((2+4+6+8)/4=20/4=5\). Thus \(\widehat{\beta}_0=\log(3)=1.0986\) and \(\widehat{\beta}_1=\log(5/3)=0.5108\), to four decimals. The group-1 mean is \(\exp(\widehat{\beta}_0+\widehat{\beta}_1)=5\).
Model-based covariance. With covariate rows \((1,0)\) for group 0 and \((1,1)\) for group 1, the information matrix is
The model-based standard errors are therefore \(\sqrt{1/12}=0.2887\) for the intercept and \(\sqrt{2/15}=0.3651\) for the group coefficient, rounded to four decimals.
HC0 covariance. In each group, the residuals around that group’s fitted mean are \(-3,-1,1,3\), with squared sum \(9+1+1+9=20\). It is convenient first to express the two fitted log means as \(\theta_0=\beta_0\) and \(\theta_1=\beta_0+\beta_1\). Each group-specific log mean has HC0 variance equal to its residual-square sum divided by the square of its total fitted mean. Therefore,
The groups contain distinct independent observations, so the estimated covariance between these two group log means is zero in this saturated two-group fit. Since \(\beta_0=\theta_0\) and \(\beta_1=\theta_1-\theta_0\), the HC0 variances are
The corresponding standard errors are \(\sqrt{5/36}=0.3727\) for the intercept and \(\sqrt{17/90}=0.4346\) for the group coefficient, rounded to four decimals. In this example, both HC0 standard errors exceed their model-based counterparts.
Check and common mistake. The group coefficient compares group 1 with group 0, so its variance includes the uncertainty in both group log means. Using only \(20/20^2\) would be the HC0 variance for the group-1 log mean \(\theta_1\), not for the contrast \(\beta_1=\theta_1-\theta_0\). The required variance is \(20/20^2+20/12^2=17/90\).
Interpreting and Using the Two Estimates
The conventional standard error is appropriate when the Poisson conditional mean and variance assumptions are suitable: conditional on the covariates and exposure, counts are independent Poisson variables with mean and variance \(\mu_i\). The model-based calculation uses the fitted means to estimate the information. If the mean model is correct but the variance differs from the Poisson assumption, HC0 provides a sandwich estimate of coefficient covariance under the stated independence and large-sample conditions.
An HC0 standard error is not guaranteed to be larger. It depends on the residuals and their association with the covariates. In the intercept-only examples, the bread was 16 in both cases, but the residual-square sums were 20 and 40. In a model with several coefficients, the matrix products also reflect which observations have residuals in which covariate patterns. Therefore, “overdispersion means multiply every standard error by a fixed factor” is not generally the HC0 calculation.
HC0 is a particular estimator, not a synonym for every adjusted standard error. Small-sample variants such as HC1 apply an additional correction, and leverage-adjusted variants alter residual contributions. A Pearson-based scale adjustment is another approach and is not identical to HC0. When a problem specifies HC0, use unadjusted squared residuals in \(B\), with no degrees-of-freedom or leverage correction.
Find \(\widehat{\beta}\) and each fitted mean \(\widehat{\mu}_i\), respecting any exposure offset.
Form \(A=\sum_i\widehat{\mu}_i x_i x_i^\mathsf{T}\). The model-based covariance is \(A^{-1}\).
Form \(B=\sum_i(Y_i-\widehat{\mu}_i)^2x_i x_i^\mathsf{T}\), using the independent observation as the unit.
Compute \(A^{-1}BA^{-1}\), take square roots of its diagonal entries, and compare them with the model-based standard errors.
The coefficient estimates and fitted rates remain unchanged when only the covariance estimator changes. What changes are standard errors and resulting uncertainty measures that use them. For ratemaking analysis, this distinction helps separate a chosen frequency indication from the strength of evidence supporting its fitted coefficients. A robust standard error does not, by itself, establish that the chosen rating structure or conditional mean is adequate.
Check Your Understanding
For each problem, calculate the requested quantities and compare the two standard errors.
- A hypothetical set of four independent unit-exposure policies has counts \(1,2,3,4\). For an intercept-only Poisson model, calculate the fitted rate and the model-based and HC0 standard errors for its log rate. (Answer: fitted rate \(2.5\); model-based SE \(0.3162\); HC0 SE \(0.2236\).)
- Four independent unit-exposure policies have counts \(1,3,5,7\). Find the intercept-only fitted rate and both standard errors for the log rate. Show the residual-square sum used for HC0. (Answer: fitted rate \(4\); residual-square sum \(20\); model-based SE \(0.2500\); HC0 SE \(0.2795\).)
- Four independent unit-exposure policies have counts \(0,2,6,8\). Calculate the fitted rate and the two log-rate standard errors, and state which estimate is larger. (Answer: fitted rate \(4\); model-based SE \(0.2500\); HC0 SE \(0.3953\); HC0 is larger.)
- A hypothetical portfolio has four independent policies with exposures \(1,1,2,2\) and counts \(1,3,4,8\). Fit an intercept-only Poisson rate and calculate model-based and HC0 standard errors for the log rate. Use policy rows as the independent units. (Answer: fitted rate \(8/3\); model-based SE \(0.2500\); HC0 SE \(0.2145\).)