Likelihood and Score With a Fixed Offset

Why an Offset Matters to the Likelihood

An offset is a known contribution to the linear predictor, not an additional coefficient to estimate. That distinction is simple to state but important when deriving likelihood-based quantities: the offset changes the predictor at which the response distribution is evaluated, while its derivative with respect to the unknown coefficients is zero. Consequently, it can change the score’s numerical value and the information, even though it does not add a new term to the coefficient derivative.

We derive these facts for independent observations from an exponential-dispersion family. The treatment keeps the link general, then shows the simpler form for a canonical link. Throughout, the design vectors and offsets are fixed, and the dispersion parameter is treated as known.

Model and Likelihood

Definition: For observations \(Y_1,\ldots,Y_n\), a GLM with a fixed offset has independent responses with \(E[Y_i]=\mu_i\), and linear predictor \(\eta_i=x_i^T\beta+o_i\). Here \(x_i\) is a fixed \(p\)-vector of covariates, \(\beta\) is the unknown coefficient vector, and \(o_i\) is a known, fixed offset. The link relation is \(g(\mu_i)=\eta_i\).

Assume the response distribution belongs to a regular exponential-dispersion family with density or mass function \(f(y_i;\theta_i,\phi)=\exp\{[y_i\theta_i-b(\theta_i)]/a(\phi)+c(y_i,\phi)\}\). The functions \(a,b,c\) and the dispersion \(\phi\) are known for this derivation. The support does not depend on the parameters, and differentiation of the log-likelihood is valid. In particular, \(b\) is twice differentiable on the parameter space, \(b''(\theta_i)>0\), and the link is differentiable with the derivatives used below existing at the fitted means.

E[Y_i]=\mu_i=b'(\theta_i), \qquad \operatorname{Var}(Y_i)=a(\phi)b''(\theta_i)=a(\phi)V(\mu_i).

Thus \(V(\mu)=b''(\theta(\mu))\) is the variance function without the dispersion factor. The inverse link gives \(\mu_i=g^{-1}(\eta_i)\), and the corresponding canonical parameter is \(\theta_i=\theta(\mu_i)\). Independence makes the log-likelihood a sum of observation-specific terms:

\ell(\beta)=\sum_{i=1}^n \left\{ \frac{y_i\theta_i-b(\theta_i)}{a(\phi)} +c(y_i,\phi) \right\}, \qquad \eta_i=x_i^T\beta+o_i.

The offset appears through \(\eta_i\), and therefore through \(\mu_i\) and \(\theta_i\). The term \(c(y_i,\phi)\) does not depend on \(\beta\) under the assumptions above. The likelihood itself, including its parameter-dependent part, generally changes when the offsets change.

Worked Example: Poisson Likelihood With a Fixed Offset

Let \(Y_1,Y_2\) be independent Poisson observations with log link, an intercept \(\beta\), and fixed offsets \(o_1=\log 2\) and \(o_2=0\). Observe \(y_1=3\) and \(y_2=0\). The means are \(\mu_i=\exp(\beta+o_i)\).

At \(\beta=0\), the predictors are \(\eta_1=\log 2\) and \(\eta_2=0\), so the means are \(\mu_1=2\) and \(\mu_2=1\). The Poisson log-likelihood, including its data-only terms, is

\ell(\beta)= 3(\beta+\log 2)-e^{\beta+\log 2}-\log(3!) +0(\beta)-e^\beta-\log(0!).

Since \(\log(3!)=\log 6\) and \(\log(0!)=0\), its value at \(\beta=0\) is \(\ell(0)=3\log 2-3-\log 6=\log(4/3)-3\), approximately \(-2.7123\).

For a Poisson log-link model the score is \(\sum_i(y_i-\mu_i)\). Hence \(U(0)=(3-2)+(0-1)=0\). The expected information is \(\sum_i\mu_i=2+1=3\), so the observed information is also \(3\) here. The offset changes the two means to \(2\) and \(1\); it does not change \(d\eta_i/d\beta=1\).

Score and Information for a General Link

Write \(h(\eta)=d\mu/d\eta\), with the derivative evaluated at the relevant predictor. From \(b'(\theta)=\mu\) and \(b''(\theta)=V(\mu)\), the chain rule gives \(d\theta/d\mu=1/V(\mu)\), and therefore \(d\theta/d\eta=h(\eta)/V(\mu)\). Since \(d\eta_i/d\beta=x_i\), differentiation of the likelihood yields the score.

Theorem: Under the model and differentiability assumptions stated above, the score for \(\beta\) is

U(\beta)=\frac{\partial \ell}{\partial\beta} =\sum_{i=1}^n x_i \frac{(y_i-\mu_i)h(\eta_i)} {a(\phi)V(\mu_i)}.

To verify the expression, first differentiate one log-likelihood contribution with respect to its canonical parameter: \(d\ell_i/d\theta_i=(y_i-b'(\theta_i))/a(\phi)=(y_i-\mu_i)/a(\phi)\). Multiplying by \(d\theta_i/d\eta_i=h(\eta_i)/V(\mu_i)\) and by \(d\eta_i/d\beta=x_i\) gives the \(i\)th summand. The known offset has derivative zero, so it does not appear as a separate factor. It does, however, determine the value of \(\eta_i\) at which \(\mu_i\), \(h(\eta_i)\), and \(V(\mu_i)\) are evaluated.

For the observed information, define \(q(\eta)=h(\eta)/V(\mu)\), where \(\mu=g^{-1}(\eta)\). Differentiating the score once more gives the negative Hessian:

J(\beta)=-\frac{\partial^2\ell}{\partial\beta\,\partial\beta^T} =\sum_{i=1}^n x_ix_i^T \left\{ \frac{h(\eta_i)^2}{a(\phi)V(\mu_i)} -\frac{y_i-\mu_i}{a(\phi)}q'(\eta_i) \right\}.

Indeed, the derivative with respect to \(\eta_i\) of \((y_i-\mu_i)q(\eta_i)/a(\phi)\) is \(\{ -h(\eta_i)q(\eta_i)+(y_i-\mu_i)q'(\eta_i)\}/a(\phi)\). Since \(h q=h^2/V\), taking the negative gives the displayed result. This observed information depends on the observed response through the residual term as well as on the offset through the predictor.

The expected information follows by taking expectation at the true parameter. Because \(E[Y_i-\mu_i]=0\), the residual term in the observed information has expectation zero. Equivalently, it follows from the variance of the score and independence. Thus:

Result: The expected Fisher information for \(\beta\) is

\mathcal I(\beta)=E[J(\beta)] =\sum_{i=1}^n x_ix_i^T \frac{h(\eta_i)^2}{a(\phi)V(\mu_i)}.

For the score-variance verification, the \(i\)th score contribution is \(x_i(Y_i-\mu_i)q(\eta_i)/a(\phi)\). Its covariance matrix is \(x_ix_i^T\operatorname{Var}(Y_i)q(\eta_i)^2/a(\phi)^2\), which equals \(x_ix_i^T h(\eta_i)^2/[a(\phi)V(\mu_i)]\), using \(\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)\). Independence makes the covariance of the total score the sum of these matrices. This establishes the information formula without requiring an unverified interchange of differentiation and integration.

Canonical Links and the Offset

With a canonical link, the link maps the mean to the canonical parameter, so \(\theta_i=\eta_i\). Then \(d\theta_i/d\eta_i=1\), equivalently \(h(\eta_i)/V(\mu_i)=1\). The score and observed information simplify, but their evaluation still uses the offset-shifted predictor.

Result: For a canonical link,

U(\beta)=\frac{1}{a(\phi)} \sum_{i=1}^n x_i(y_i-\mu_i), \qquad J(\beta)=\frac{1}{a(\phi)} \sum_{i=1}^n x_ix_i^T V(\mu_i), \qquad \mathcal I(\beta)=J(\beta).

To derive the observed information, differentiate the canonical score. Since \(d\mu_i/d\eta_i=V(\mu_i)\), and \(d\eta_i/d\beta=x_i\), differentiating the \(i\)th score contribution gives \(-x_ix_i^TV(\mu_i)/a(\phi)\). Its negative is the stated information contribution. The equality of observed and expected information follows because this expression has no residual term.

Worked Example: Bernoulli Logit Score and Information

Suppose \(Y_1,Y_2\) are independent Bernoulli observations with a common intercept \(\beta\), logit link, and fixed offsets \(o_1=0\) and \(o_2=\log 2\). Let the observed outcomes be \(y_1=1,y_2=0\). The success probabilities are \(p_i=\exp(\eta_i)/(1+\exp(\eta_i))\), with \(\eta_i=\beta+o_i\).

At \(\beta=0\), \(p_1=1/2\) and \(p_2=2/3\). The Bernoulli log-likelihood is \(\ell=\sum_i\{y_i\eta_i-\log(1+\exp(\eta_i))\}\), so \(\ell(0)=-\log 2-\log 3=-\log 6\).

The logit link is canonical for the Bernoulli family. Therefore the score is \(U(0)=(1-1/2)+(0-2/3)=-1/6\). The observed and expected information are both \(p_1(1-p_1)+p_2(1-p_2)=1/4+2/9=17/36\).

The offsets alter both probabilities and hence the numerical likelihood, score, and information. But \(d\eta_1/d\beta=d\eta_2/d\beta=1\). No derivative of either known offset is added.

Worked Example: Probit Information With Two Fixed Offsets

Consider two independent Bernoulli observations with an intercept \(\beta\), probit link \(p_i=\Phi(\eta_i)\), and known offsets \(o_1=0,o_2=1\). Evaluate expected information at \(\beta=0\). For this link, \(h(\eta)=\varphi(\eta)\), where \(\varphi\) and \(\Phi\) are the standard normal density and distribution functions, and \(V(p)=p(1-p)\).

Each design value is \(x_i=1\). The general expected-information formula gives a contribution \(\varphi(\eta_i)^2/[\Phi(\eta_i)\{1-\Phi(\eta_i)\}]\). For the first observation, \(\eta_1=0\), \(\varphi(0)=1/\sqrt{2\pi}\), and \(\Phi(0)=1/2\), so the contribution is \((1/(2\pi))/(1/4)=2/\pi\), approximately \(0.636620\).

For the second observation, \(\eta_2=1\), \(\varphi(1)\) is approximately \(0.241971\), and \(\Phi(1)\) is approximately \(0.841345\). Therefore its contribution is \[ \frac{0.241971^2}{0.841345(1-0.841345)} \approx 0.438629. \] The total expected information for the intercept is approximately \(0.636620+0.438629=1.075249\). These are noncanonical-link information weights, evaluated at predictors that include the fixed offsets.

What Changes, and What Does Not

A fixed offset is not invisible to inference. Changing its value changes the predictor, the implied mean, the likelihood contribution, and generally the score and information evaluated at a given \(\beta\). In a noncanonical model it can also change both the derivative factor \(h(\eta_i)/V(\mu_i)\) and the residual-dependent part of the observed information.

What remains unchanged is the coefficient derivative of the linear predictor: \(\partial\eta_i/\partial\beta=x_i\). There is no coefficient associated with the offset, and no offset derivative enters the score. This is why the right distinction is not “offsets affect derivatives” versus “offsets do not affect derivatives.” They affect the values at which derivatives are evaluated, but they are not themselves differentiated with respect to \(\beta\).

Likelihood: changes through the offset-shifted \(\eta_i,\mu_i,\theta_i\).
Coefficient chain rule: remains \(d\eta_i/d\beta=x_i\).
Score and information values: generally change because their mean- and link-dependent factors are evaluated at the shifted predictor.
New estimated coefficient: none is introduced by a fixed offset.

Common mistake. A tempting but incorrect argument is: “The offset is fixed, so it cannot affect the score or information.” It is true that its derivative with respect to \(\beta\) is zero. But the score contains \(\mu_i=g^{-1}(x_i^T\beta+o_i)\), and the information contains functions of that same predictor. The correction is to set the offset derivative to zero while retaining the offset in every predictor-dependent likelihood quantity.

A second error is to treat an offset as if it were an estimated intercept. Doing so would add a derivative direction that the model does not contain. The model’s coefficient vector is \(\beta\), and differentiation is with respect to its entries only; the known offset remains fixed throughout.

Check Your Understanding

Use the definitions and formulas above to answer the following questions.

In a GLM with \(\eta_i=x_i^T\beta+o_i\), what is \(\partial\eta_i/\partial\beta\), and why?
For a noncanonical link, identify the predictor-dependent factors in the score that can change when a fixed offset changes.
Why does the residual term in the observed information vanish after taking expectation?
For a canonical link, what are the score and expected information in terms of \(x_i,y_i,\mu_i,V(\mu_i)\), and \(a(\phi)\)?
Explain why “the offset is fixed” does not imply that the score is numerically unaffected by the offset.