The Poisson Score As An Observed–Fitted Balance

From the Poisson Likelihood to a Balance Equation

In Poisson regression, each fitted mean is linked to a covariate profile, so the model does not generally fit every observed count exactly. Instead, at an interior maximum of the likelihood, the score imposes a set of aggregate constraints: observed-minus-fitted counts must balance after weighting by each covariate. These constraints are useful for interpreting the fitted model, but they are not statements that every residual is zero.

Assume \(Y_1,\ldots,Y_n\) are independent responses, with fixed covariate vectors \(x_i\in\mathbb{R}^p\), and that \(Y_i\) has a Poisson distribution with mean \(\mu_i(\beta)=\exp(x_i^\mathsf{T}\beta)\). Here \(\beta\in\mathbb{R}^p\) is unrestricted, and \(y_i\) denotes the observed count. Write \(X\) for the \(n\times p\) design matrix whose \(i\)th row is \(x_i^\mathsf{T}\), and write \(y\) and \(\mu(\beta)\) for the corresponding \(n\)-vectors.

Definition: The score vector for \(\beta\) is the gradient of the log-likelihood with respect to \(\beta\), denoted \(U(\beta)=\nabla_\beta\ell(\beta)\). The fitted mean at observation \(i\), for a specified coefficient vector \(\beta\), is \(\mu_i(\beta)=\exp(x_i^\mathsf{T}\beta)\). At an estimated coefficient vector \(\hat\beta\), write \(\hat\mu_i=\mu_i(\hat\beta)\).

Deriving the Poisson Score

The Poisson probability mass function gives \(P(Y_i=y_i)=\exp(-\mu_i)\mu_i^{y_i}/y_i!\). Independence makes the likelihood a product of these probabilities. Substituting the log-link mean and taking logarithms gives

\ell(\beta) =\sum_{i=1}^n\left\{y_i x_i^\mathsf{T}\beta-\exp(x_i^\mathsf{T}\beta)-\log(y_i!)\right\}.

The log-likelihood terms are not weighted residuals. Rather, differentiating them produces the score, which combines observed-minus-fitted counts using the covariate vectors as weights.

Result: For the Poisson regression model with log link,

U(\beta) =\sum_{i=1}^n x_i\{y_i-\mu_i(\beta)\} =X^\mathsf{T}\{y-\mu(\beta)\}.

To verify the formula, consider any direction \(a\in\mathbb{R}^p\) and differentiate the log-likelihood along the line \(\beta+t a\) at \(t=0\). Since the derivative of \(\exp(x_i^\mathsf{T}(\beta+t a))\) at zero is \(\mu_i(\beta)x_i^\mathsf{T}a\), we obtain

\left.\frac{d}{dt}\ell(\beta+t a)\right|_{t=0} =\sum_{i=1}^n \{y_i-\mu_i(\beta)\}x_i^\mathsf{T}a =a^\mathsf{T}\sum_{i=1}^n x_i\{y_i-\mu_i(\beta)\}.

By the definition of the gradient, the directional derivative is also \(a^\mathsf{T}\nabla_\beta\ell(\beta)\). As this equality holds for every direction \(a\), the gradient must equal the displayed sum. Placing the \(x_i^\mathsf{T}\) vectors in the rows of \(X\) gives the matrix expression.

Why an Interior Maximum Gives Balance

The score is a derivative, so its value at an interior maximum is constrained by the first-order condition. Here “interior” means that the maximizing coefficient vector lies in the open parameter space \(\mathbb{R}^p\), rather than on a boundary imposed by parameter restrictions.

Theorem: If the Poisson log-likelihood is differentiable on \(\mathbb{R}^p\) and \(\hat\beta\in\mathbb{R}^p\) is a local maximum, then \(U(\hat\beta)=0\). Consequently,

X^\mathsf{T}(y-\hat\mu)=0, \qquad \sum_{i=1}^n x_i(y_i-\hat\mu_i)=0.

Proof. Fix any direction \(a\in\mathbb{R}^p\). Because \(\hat\beta\) is interior, \(\hat\beta+t a\) remains in the parameter space for all sufficiently small positive and negative \(t\). The one-variable function \(g(t)=\ell(\hat\beta+t a)\) therefore has a local maximum at \(t=0\). Differentiability implies \(g'(0)=0\). By the directional-derivative identity,

0=g'(0)=a^\mathsf{T}U(\hat\beta).

This holds for every \(a\). In particular, taking \(a\) to be each coordinate vector shows that every component of \(U(\hat\beta)\) is zero. Substituting the score formula yields the balance equation. \(\square\)

For coordinate \(j\), the equation says that the residuals \(y_i-\hat\mu_i\), weighted by the \(j\)th covariate \(x_{ij}\), sum to zero. If the model includes an intercept, its design-matrix column consists of ones, so the corresponding equation is

\sum_{i=1}^n(y_i-\hat\mu_i)=0, \qquad\text{or equivalently}\qquad \sum_{i=1}^n y_i=\sum_{i=1}^n\hat\mu_i.

Thus the total observed count equals the total fitted count at an interior maximum with an intercept. Without an intercept column, this particular total-count equation need not hold. And even with an intercept, the equation concerns the total: it does not require \(y_i=\hat\mu_i\) observation by observation.

Worked Example: An Intercept-Only Model

Suppose four independent Poisson counts are \(1,2,3,2\), and the model has only an intercept: \(\log(\mu_i)=\beta_0\). Find the fitted mean that satisfies the score equation and check the balance.

The intercept-only model gives the same mean \(\mu=\exp(\beta_0)\) for all four observations. Its score is the sum of the four residuals:

U(\beta_0)=\sum_{i=1}^4(y_i-\mu) =(1+2+3+2)-4\mu=8-4\mu.

Setting the score to zero gives \(\hat\mu=2\), so \(\hat\beta_0=\log 2\). The fitted counts are \(2,2,2,2\), and their residuals are \(-1,0,1,0\). Their sum is \(-1+0+1+0=0\), as required. The individual residuals are not all zero, even though the intercept balance holds.

Each Covariate Has Its Own Balance

The intercept equation balances counts without weights. Other score equations use the corresponding covariate values as weights. A covariate that is zero for some observations contributes no residual from those observations to its balance equation; a large covariate value gives its residual a correspondingly large weight. The signs matter as well: positive and negative weighted residuals can cancel.

Worked Example: Balance Across Two Covariate Groups

Suppose a binary predictor \(x_i\) is zero for two observations with counts \(1,3\), and one for three observations with counts \(2,4,6\). The model is \(\log(\mu_i)=\beta_0+\beta_1x_i\). Verify the score balance at the coefficients that fit the two group means.

For \(x=0\), the mean count is \((1+3)/2=2\). For \(x=1\), it is \((2+4+6)/3=4\). The model means are \(\mu_0=\exp(\beta_0)\) and \(\mu_1=\exp(\beta_0+\beta_1)\). Taking \(\hat\beta_0=\log 2\) and \(\hat\beta_1=\log 2\) gives \(\hat\mu_0=2\) and \(\hat\mu_1=4\).

The residuals in the \(x=0\) group are \(-1,1\), and those in the \(x=1\) group are \(-2,0,2\). The intercept score is the sum of all residuals:

$$ (-1+1)+(-2+0+2)=0. $$

The slope score weights each residual by \(x_i\). The \(x=0\) observations contribute zero, and the other group contributes

$$ 0(-1+1)+1(-2+0+2)=0. $$

In this example the two group residual sums happen to vanish separately, because the fitted group means equal the observed group means. That group-by-group property is special to this binary-predictor model; score balance in a general model need not imply balance within every covariate group.

Worked Example: Weighted Balance Without Individual Fits

Take three predictor values \(x=(-1,0,1)\), include an intercept, and observe counts \(y=(2,0,5)\). Evaluate the score at \(\beta_0=\log 2,\ \beta_1=\log 2\), where \(\log(\mu_i)=\beta_0+\beta_1x_i\).

The fitted means, in the same order as the observations, are

\hat\mu =\left(\exp(\log 2-\log 2),\ \exp(\log 2),\ \exp(\log 2+\log 2)\right) =(1,2,4).

Thus \(y-\hat\mu=(1,-2,1)\). The intercept score is \(1-2+1=0\), while the slope score is

$$ (-1)(1)+(0)(-2)+(1)(1)=0. $$

Both score components vanish, although none of the three observed counts equals its fitted mean. This calculation illustrates the equations at the specified coefficient vector; a zero score is a first-order condition and, by itself, does not establish that a point is a maximum.

What the Balance Does—and Does Not—Say

The balance equations summarize how the residual vector relates to the columns of the design matrix. If a column represents an intercept, the residuals sum to zero. If a column represents a predictor, its predictor-weighted residuals sum to zero. With several predictors, each column has its own equation. These are simultaneous constraints on the residual vector, not a requirement that all residuals vanish.

The score is important because it connects the likelihood calculation to the fitted model’s covariate structure. It also provides equations that an interior maximum must satisfy. But these equations alone do not prove that a solution is a maximum, that it is unique, or that a finite maximum exists. They are necessary first-order conditions under the stated assumptions.

Common Mistake

Wrong reasoning: “The likelihood is a predictor-weighted sum of observed-minus-fitted counts.” This confuses the log-likelihood with its derivative. The log-likelihood is a sum of terms \(y_i x_i^\mathsf{T}\beta-\exp(x_i^\mathsf{T}\beta)-\log(y_i!)\); differentiating it produces the covariate-weighted residual sum.
Correction: “At an interior maximum, the score—the gradient of the log-likelihood—is the predictor-weighted sum of observed-minus-fitted counts, and that score equals zero.”
Wrong reasoning: “A zero score means every observation is fitted exactly.” The score equations require weighted sums to vanish, so residuals can cancel.
Correction: Check each column’s weighted balance equation. Do not infer that \(y_i=\hat\mu_i\) for every \(i\).

Key takeaway: In Poisson regression with a log link, \(U(\beta)=X^\mathsf{T}\{y-\mu(\beta)\}\). At an interior maximum, each covariate-weighted residual sum is zero. An intercept therefore balances total observed and fitted counts, but the balance does not require exact fits for individual observations.

Check Your Understanding

Use the score formula and its balance interpretation to answer the following.

For an interior maximum in a Poisson regression with an intercept, what equality holds between the total observed and total fitted counts?
A predictor column has values \(0,0,1,1\). Write the score-balance equation for its coefficient.
Why does a zero score not imply that every observed count equals its fitted mean?
If a model has no intercept, which part of the usual total-count balance argument no longer applies?
Explain why it is the score, rather than the log-likelihood itself, that is a covariate-weighted sum of observed-minus-fitted counts.