Conditional Means Versus Future Counts

Two Different Questions at One Covariate Profile

Suppose a Poisson regression model is used to describe a count for a subject with covariate profile \(x_0\). There are two distinct questions. First, how uncertain are we about the model’s conditional expected count at \(x_0\), given that the coefficients have been estimated from data? Second, how much can a future count for a subject at that same profile vary around the true conditional mean? The first concerns estimation across repeated datasets; the second concerns the response distribution for a future observation.

These questions share the same mean but do not have the same answer. In particular, a confidence interval for a fitted mean is not a range containing 95% of future counts. Conversely, the Poisson standard deviation of a future count does not measure how precisely the regression coefficients, or the fitted mean, have been estimated.

We hold the covariate profile fixed throughout. Write the Poisson log-mean model as \(\log\{\mu(x)\}=x^\mathsf{T}\beta\), where \(\mu(x)=E[Y\mid x]\), and assume \(Y\mid x\sim\operatorname{Poisson}(\mu(x))\). Here \(\beta\) is the model parameter and \(x_0\) is the specified profile. The true \(\mu(x_0)\) is fixed under the model, though unknown to the analyst.

What the Poisson Model Says About a Future Count

For a Poisson response with mean \(\mu>0\), the probability of count \(k\) is \(P(Y=k)=e^{-\mu}\mu^k/k!\), for \(k=0,1,2,\ldots\). The mean and variance are both \(\mu\). Thus, at a fixed profile, the model’s conditional standard deviation for a future count is \(\sqrt{\mu(x_0)}\).

For completeness, the moment calculation follows directly from the Poisson probability mass function. For the mean, use \(k/k!=1/(k-1)!\) for \(k\geq1\):

Similarly, since \(k(k-1)/k!=1/(k-2)!\) for \(k\geq2\),

Because \(Y^2=Y(Y-1)+Y\), it follows that \(E[Y^2\mid x_0]=\mu^2+\mu\), and hence

This result concerns the response distribution at a fixed value of the model parameter. If \(\mu(x_0)\) is not known, substituting \(\hat{\mu}(x_0)\) gives a plug-in estimate of the future-count variance, not a calculation of uncertainty in \(\hat{\mu}(x_0)\).

Uncertainty in the Estimated Conditional Mean

Let \(\hat{\beta}\) be an estimate of \(\beta\), and write \(\hat{\eta}(x_0)=x_0^\mathsf{T}\hat{\beta}\) and \(\hat{\mu}(x_0)=\exp\{\hat{\eta}(x_0)\}\). The estimated mean changes across repeated samples because \(\hat{\beta}\) changes. Its uncertainty is commonly summarized by a standard error or a confidence interval based on the coefficient estimate and its estimated covariance matrix.

For example, suppose the fitted linear predictor has estimated standard error \(s_{\hat{\eta}}\). A first-order Taylor expansion of the exponential function gives the familiar approximation

This approximation is for repeated-sample variation in the estimated mean. To see why, suppose \(\hat{\eta}(x_0)\) is consistent for \(\eta(x_0)\), and is asymptotically normal with variance approximately \(s_{\hat{\eta}}^2\). For \(g(u)=e^u\), the derivative is \(g'(u)=e^u\). A first-order expansion around \(\eta(x_0)\) is

where the remainder \(r_n\) is of smaller order than \(\hat{\eta}(x_0)-\eta(x_0)\) under the usual conditions for the first-order delta-method approximation. Thus the leading random part of the fitted mean is approximately \(\mu(x_0)\{\hat{\eta}(x_0)-\eta(x_0)\}\). Its variance is approximately \(\mu(x_0)^2s_{\hat{\eta}}^2\); replacing the unknown mean by its estimate yields the stated standard-error approximation. This calculation concerns the estimator, not the distribution of a future \(Y\).

The exponential transformation in this example preserves the confidence level because it is strictly increasing: an interval containing \(\eta(x_0)\) contains its exponential after both endpoints are exponentiated, exactly when the original interval contains \(\eta(x_0)\). This gives an interval on the positive mean scale. It does not turn the interval into a range for future observations.

One Profile, Two Calculations

When the estimated coefficient covariance matrix is available, uncertainty in the fitted log mean at a profile can be calculated from that matrix. If \(\widehat{\operatorname{Var}}(\hat{\beta})=V\), then the estimated variance of \(x_0^\mathsf{T}\hat{\beta}\) is \(x_0^\mathsf{T}Vx_0\). The Poisson calculation is separate: it uses the conditional mean to estimate the response variance. The following example performs both calculations for the same profile.

How to Read the Numbers Correctly

The distinction matters whenever a result is reported as a prediction. A statement such as “the expected count is about 2.2” identifies a conditional mean. A confidence interval for that mean describes uncertainty from fitting the regression model. A future observation is a count, and under the Poisson model it varies around the true mean according to its conditional response distribution. A mean estimate and its confidence interval do not, by themselves, describe that full response variation.

A quick check is to ask what would change if the data used to estimate the model were collected again, versus what would change if a different future subject were observed at the same profile. Changes in the fitted coefficients and fitted mean belong to estimation uncertainty. Changes in the realized count, with the profile and true model held fixed, belong to future-count variation.

Common Mistake

A common error is to calculate \(\sqrt{\hat{\mu}(x_0)}\) and call it the standard error of the estimated mean. That quantity follows from the Poisson response variance and estimates the standard deviation of a future count at the profile. It does not account for how much \(\hat{\beta}\), and therefore \(\hat{\mu}(x_0)\), would change across repeated fitted datasets.

The reverse error is also possible: using the standard error of \(\hat{\mu}(x_0)\) as if it measured the spread of future counts. That omits the response variation specified by the Poisson model. Before interpreting any uncertainty number, identify its target: the unknown conditional mean, or a future response conditional on the profile.