Weighted Likelihood Scores and Information

Why Replication Changes the Score

A frequency weight records how many times an observation is represented in a data set. If a row has weight \(r_i=4\), its likelihood contribution is counted four times, just as if the row had been entered four times. This changes the relative contribution of the row to the score and information. The key is to keep the sampling interpretation clear: four independent replicated observations are not the same random object as one response whose score has simply been multiplied by four.

We derive the score equations and expected information for frequency-replicated observations, then use examples to show how unequal and zero weights affect a fit. The distinction between score covariance and expected information matters especially when writing a weighted score using one response per row.

Definition: Suppose there are \(m\) distinct design rows, with row \(i\) having covariate vector \(x_i\) and integer frequency \(r_i\geq 0\). Frequency replication means that row \(i\) represents \(r_i\) independent responses from the same GLM distribution at that design row. The expanded data have responses \(Y_{ij}\), for \(j=1,\ldots,r_i\). A row with \(r_i=0\) contributes no observations.

For the derivations, assume a regular one-parameter exponential dispersion family with density or mass function

f(y;\theta,\phi) = \exp\left\{ \frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi) \right\}.

Here \(\phi\) is fixed, the support does not depend on the model parameters, and differentiation under the expectation is valid. The mean and variance are \( \mu=b'(\theta) \) and \( \operatorname{Var}(Y)=a(\phi)V(\mu) \), respectively. Let \( \eta_i=x_i^\mathsf{T}\beta \), with a differentiable link \(g(\mu_i)=\eta_i\); assume the derivatives used below exist and \(V(\mu_i)>0\). A fixed offset can be included in \(\eta_i\) without changing the derivatives with respect to \(\beta\).

Weighted Likelihood and Score Equations

The expanded log-likelihood is a sum over all represented observations. If the replicated responses in row \(i\) have the same observed value \(y_i\), this sum is exactly the row-frequency log-likelihood \( \ell_w(\beta)=\sum_i r_i\ell_i(y_i;\beta) \), up to terms constant in \(\beta\). The same score formula can be written with individual copy responses when they differ.

\ell_{\mathrm{rep}}(\beta) = \sum_{i=1}^{m}\sum_{j=1}^{r_i} \log f(Y_{ij};\theta_i,\phi), \qquad \ell_w(\beta) = \sum_{i=1}^{m}r_i\log f(y_i;\theta_i,\phi).

The second expression is the compressed, frequency-weighted form when the row’s observed value is repeated. To differentiate, use \(d\theta_i/d\mu_i=1/V(\mu_i)\), which follows from \(d\mu_i/d\theta_i=V(\mu_i)\). Also, \(d\mu_i/d\beta=(d\mu_i/d\eta_i)x_i\).

Result: For the compressed frequency-weighted likelihood, the score equation is

U_w(\beta) = \frac{\partial\ell_w(\beta)}{\partial\beta} = \sum_{i=1}^{m} r_i x_i \frac{y_i-\mu_i}{a(\phi)} \frac{d\theta_i}{d\eta_i} = \sum_{i=1}^{m} r_i x_i \frac{y_i-\mu_i}{a(\phi)V(\mu_i)} \frac{d\mu_i}{d\eta_i} =0.

For the expanded data, replace \(r_i(y_i-\mu_i)\) in this expression by \(\sum_{j=1}^{r_i}(Y_{ij}-\mu_i)\). When all the observed copies in a row have the same response, the expanded score reduces to the displayed row-weighted score. The equation need not have a closed-form solution; it states the condition a finite interior maximum must satisfy.

Worked Example: A Poisson Fit With Unequal Frequencies

Consider a Poisson GLM with log link and two design rows, \(x_1=(1,0)^\mathsf{T}\) and \(x_2=(1,1)^\mathsf{T}\). The observed counts are \(y_1=2\), \(y_2=1\), with frequencies \(r_1=3\), \(r_2=1\). Write \(\beta=(\alpha,\beta_1)^\mathsf{T}\), so \(\mu_1=e^\alpha\) and \(\mu_2=e^{\alpha+\beta_1}\). For a Poisson GLM, \(a(\phi)=1\), \(V(\mu)=\mu\), and \(d\mu/d\eta=\mu\); hence each score contribution is \(r_i x_i(y_i-\mu_i)\).

The intercept and slope equations are therefore

3(2-\mu_1)+(1-\mu_2)=0, \qquad 1(1-\mu_2)=0.

The slope equation gives \(\mu_2=1\). Substituting into the intercept equation gives \(3(2-\mu_1)=0\), so \(\mu_1=2\). Thus \( \hat\alpha=\log 2 \) and \( \hat\beta_1=\log(1/2) \). The weighted likelihood fits the two group means, while the first row contributes three times as much to the intercept equation as it would at frequency one.

The expected information at this fit is \( \sum_i r_i\mu_i x_ix_i^\mathsf{T} \). The two rows contribute \(6\begin{pmatrix}1&0\\0&0\end{pmatrix}\) and \(1\begin{pmatrix}1&1\\1&1\end{pmatrix}\), respectively, giving

\mathcal I_w(\hat\beta) = \begin{pmatrix} 7&1\\ 1&1 \end{pmatrix}.

This is the expected information under the independent-replication interpretation; it is not obtained by taking the covariance of a single response per row after multiplying each row score by its frequency.

Expected Information: Derivation and Interpretation

The expected information is the expected negative Hessian of the replicated log-likelihood. Under the stated regularity conditions, it can also be computed as the covariance of the score formed by summing the separate copy-level scores. We derive both expressions to make explicit which score is meant.

Result: For independent replicated responses, the expected information for \(\beta\) is

\mathcal I_w(\beta) = -E\left[ \frac{\partial^2\ell_{\mathrm{rep}}(\beta)} {\partial\beta\,\partial\beta^\mathsf{T}} \right] = E\left[U_{\mathrm{rep}}(\beta)U_{\mathrm{rep}}(\beta)^\mathsf{T}\right] = \sum_{i=1}^{m}r_i x_ix_i^\mathsf{T} \frac{(d\mu_i/d\eta_i)^2}{a(\phi)V(\mu_i)},

where

U_{\mathrm{rep}}(\beta) = \sum_{i=1}^{m}\sum_{j=1}^{r_i} x_i\frac{Y_{ij}-\mu_i}{a(\phi)} \frac{d\theta_i}{d\eta_i}.

Proof. For one copy in row \(i\), its score is \(s_{ij}=x_i(Y_{ij}-\mu_i)(d\theta_i/d\eta_i)/a(\phi)\). Its expectation is zero because \(E[Y_{ij}]=\mu_i\). Independence across copies makes cross-covariances zero, so

E[U_{\mathrm{rep}}U_{\mathrm{rep}}^\mathsf{T}] = \sum_i r_i x_ix_i^\mathsf{T} \frac{\operatorname{Var}(Y_{ij})}{a(\phi)^2} \left(\frac{d\theta_i}{d\eta_i}\right)^2 = \sum_i r_i x_ix_i^\mathsf{T} \frac{(d\mu_i/d\eta_i)^2}{a(\phi)V(\mu_i)}.

For the Hessian, differentiate one copy’s score with respect to \(\beta^\mathsf{T}\). The term involving \(Y_{ij}-\mu_i\) and the derivative of \(d\theta_i/d\eta_i\) has expectation zero. The remaining term is

-E\left[ \frac{\partial s_{ij}}{\partial\beta^\mathsf{T}} \right] = x_ix_i^\mathsf{T} \frac{d\mu_i}{d\eta_i} \frac{d\theta_i}{d\eta_i} \frac{1}{a(\phi)} = x_ix_i^\mathsf{T} \frac{(d\mu_i/d\eta_i)^2}{a(\phi)V(\mu_i)}.

Summing over the \(r_i\) copies gives the stated expected negative Hessian and completes the proof. \(\square\)

For a general link, the factor \(d\mu_i/d\eta_i\) remains in the information. In the canonical-link case, \(d\theta_i/d\eta_i=1\), and the formula simplifies to \( \sum_i r_i x_ix_i^\mathsf{T}V(\mu_i)/a(\phi) \). In either case, each positive-frequency row adds a positive semidefinite matrix. The information is nonsingular only when the weighted design directions, with nonzero sensitivity, span the coefficient space.

Unequal and Zero Frequencies in Practice

Worked Example: Unequal Frequencies in a Gaussian Intercept Model

Suppose \(Y\sim N(\beta,\sigma^2)\), where \(\sigma^2\) is known, and there are two observed rows: \(y_1=0\) with frequency \(r_1=3\), and \(y_2=10\) with frequency \(r_2=1\). The score for one observation is \((y-\beta)/\sigma^2\), so the frequency-weighted score equation is

\frac{3(0-\beta)+(10-\beta)}{\sigma^2}=0.

Multiplying by \(\sigma^2\) and solving gives \(-4\beta+10=0\), hence \( \hat\beta=2.5 \). This is also the mean of the four replicated observations \(0,0,0,10\). The expected information is \( (3+1)/\sigma^2=4/\sigma^2 \). If both rows instead had frequency one, the estimate would be \(5\) and the information \(2/\sigma^2\). Frequency therefore changes both the fitted value and the amount of replication-based information.

Worked Example: A Zero Frequency Removes a Coefficient Direction

Use a Poisson log-link model with \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\). Let \(y_1=2\), \(r_1=2\), and \(r_2=0\). The weighted score is

U_w(\beta) = 2x_1(2-e^\alpha)+0x_2(y_2-e^{\alpha+\beta_1}).

The intercept equation is \(2(2-e^\alpha)=0\), giving \(\hat\alpha=\log 2\). The slope equation is identically zero for every value of \(\beta_1\). The likelihood does not depend on the second row or on \(\beta_1\), so that coefficient is not identifiable from these weighted data. At \(\alpha=\log 2\), the expected information is

\mathcal I_w(\beta) = 2e^\alpha x_1x_1^\mathsf{T} = \begin{pmatrix} 4&0\\ 0&0 \end{pmatrix}.

The zero eigenvalue reflects the missing information in the slope direction. A zero-frequency row is not a row with weak evidence; it is absent from the frequency-replicated likelihood.

A Crucial Score-Covariance Distinction

The expected information formula has one factor of \(r_i\), but the covariance of a score written using one response \(Y_i\) per row and then multiplied by \(r_i\) has \(r_i^2\). Specifically, if \(U_w=\sum_i r_i s_i(Y_i)\) and the \(Y_i\) are independent single draws, then

\operatorname{Var}(U_w) = \sum_i r_i^2\operatorname{Var}(s_i(Y_i)).

That is not the score covariance for \(r_i\) independent copies. For the copies, the score is \(\sum_i\sum_j s_i(Y_{ij})\), whose covariance has \(r_i\) identical variance contributions. For instance, in a Poisson intercept model with one row, \(r=2\), and \(\mu=1\), the single-row weighted score is \(2(Y-1)\), whose variance is \(4\). The replicated-data information is \(2\), because it comes from two independent Poisson observations, each with variance \(1\). These are different random-score constructions; do not equate their covariances.

Common pitfall: It is incorrect to take the variance of the row-weighted score based on one response per row and call it the expected information for replicated data. The correction is to define the score over the independent copies when using the score-covariance identity. The expected negative Hessian of the frequency-weighted log-likelihood has one factor of \(r_i\), as does the covariance of the copy-level score.

Frequency weights also have a specific interpretation: an integer weight says how many independent copies the row represents. It is not automatically a statement that a response has greater precision for some other reason. That distinction matters when choosing a likelihood and interpreting standard errors. Unequal frequencies increase the influence of replicated rows; zero frequencies remove rows altogether. Even positive frequencies cannot guarantee that every coefficient is estimable: the nonzero-weight design must contain enough independent directions.

Check Your Understanding

Use the distinction between the weighted row score and the copy-level score in your answers.

For a Poisson GLM with log link, write the frequency-weighted score contribution for row \(i\) in terms of \(r_i\), \(x_i\), \(y_i\), and \(\mu_i\).
Why does the expected information from \(r_i\) independent copies have one factor of \(r_i\), rather than \(r_i^2\)?
In a Gaussian intercept model with known variance, what estimate results from frequencies 2 and 1 on responses 1 and 7? Show the score equation.
What does a zero frequency do to that row’s contribution to the likelihood, score, and information?
Why can a positive-frequency data set still fail to identify all regression coefficients?