Frequency Weights as Replication Counts

What a Frequency Weight Means

A frequency weight has a literal counting interpretation: it says that the displayed observation represents that many independent observations with the same recorded response and covariates. This meaning is narrower than simply saying that an observation is “more important.” It gives a precise likelihood, and therefore determines how the score and information change.

Suppose row $i$ records response $y_i$ and covariate vector $x_i$. If its frequency is $w_i=4$, the row represents four observations, each with response $y_i$ and covariates $x_i$. They are modeled as independent draws from the same response distribution at that covariate value. The frequency is a positive integer; a row with frequency zero contributes nothing and can be omitted.

Definition: An integer frequency weight $w_i$ for row $i$ is the number of independent observations represented by that row. Under the replication interpretation, the likelihood contribution is the product of $w_i$ copies of the individual-observation likelihood contribution. A frequency weight is not, by itself, a trial count or a generic measure of precision.

The independence assumption matters. Four records for four distinct people that happen to have the same response and covariates can be represented by one row with frequency four. Copying one person’s record four times does not create four independent observations. The latter inflates the likelihood without providing the information that four actual units would provide.

Write $f(y_i;\beta)$ for the response density or mass function under the GLM, with the covariates and any fixed model components understood. For independent observations, the likelihood is the product of the individual likelihood contributions. Grouping identical records into rows gives the following expression.

L_w(\beta)=\prod_{i=1}^m f(y_i;\beta)^{w_i}, \qquad \ell_w(\beta)=\sum_{i=1}^m w_i\log f(y_i;\beta).

The subscript $w$ emphasizes that this is the likelihood for the replicated data represented by the rows. Any term in the log-likelihood that depends on the observed response but not on $\beta$ is also repeated $w_i$ times. Such a term may be dropped when finding a maximum or differentiating with respect to $\beta$, but it remains part of the full log-likelihood.

How Replication Changes the Derivatives

Let $U_i(\beta)=\partial \log f(y_i;\beta)/\partial\beta$ be the score vector from one observation in row $i$. Let $H_i(\beta)$ be its Hessian matrix, the matrix of second derivatives of its log-likelihood. Differentiating the weighted log-likelihood gives exact identities: each row’s contribution to either derivative is multiplied by its replication count.

Result: Wherever the individual log-likelihood contributions are twice differentiable, $$ U_w(\beta)=\frac{\partial\ell_w(\beta)}{\partial\beta} =\sum_{i=1}^m w_i U_i(\beta), \qquad J_w(\beta)=-\frac{\partial^2\ell_w(\beta)}{\partial\beta\,\partial\beta'} =\sum_{i=1}^m w_i[-H_i(\beta)]. $$ Here $J_w(\beta)$ is the observed information matrix.

These identities need no special GLM link. They follow from the replication likelihood itself. A useful further consequence concerns Fisher information. For this result, assume that the support of each response distribution does not depend on $\beta$, that the likelihood is differentiable in a neighborhood of the parameter value of interest, and that differentiation can be interchanged with integration (or summation). Assume also the required derivatives and score second moments are finite. These regularity conditions yield the usual individual-observation information identity.

Theorem: Under those regularity conditions, the Fisher information in the replicated observations is $$ \mathcal I_w(\beta)=E[J_w(\beta)] =\sum_{i=1}^m w_i\mathcal I_i(\beta), \qquad \mathcal I_i(\beta)=E[-H_i(\beta)]. $$ If the $w_i$ represented observations in each row are independent, this is also the variance of the score for the full, ungrouped sample.

Proof. For one observation with density or mass function $f(y;\beta)$, normalization gives $\int f(y;\beta)\,dy=1$, with a sum replacing the integral for a discrete response. Differentiating once under the integral,

E[U(\beta)] =\int \frac{\partial\log f(y;\beta)}{\partial\beta}f(y;\beta)\,dy =\int \frac{\partial f(y;\beta)}{\partial\beta}\,dy =\frac{\partial}{\partial\beta}1=0.

Differentiate the normalization a second time. Since $\partial^2 f/\partial\beta\,\partial\beta' =f\{H+UU'\}$, the same regularity conditions give $E[H+UU']=0$, or $E[-H]=E[UU']$. For a replicated sample, the observed negative Hessian is the sum of $w_i$ individual negative Hessians, so taking expectations gives $\mathcal I_w=\sum_i w_i\mathcal I_i$. In the full sample the score is a sum of independent individual scores. Each has mean zero, so cross-covariances between distinct observations are zero; its variance is therefore the sum of their score variances, $\sum_i w_i E[U_iU_i']$. By the individual information identity, this is the same $\mathcal I_w$. $\square$

The variance statement is about the full independent sample, not about randomly drawing one response and then multiplying its score by $w_i$. That operation would produce a different random quantity. The frequency-weighted formula records the score contributions of the $w_i$ observations represented by the row.

GLM Formulas and Worked Examples

In a GLM, the response distribution at row $i$ depends on its linear predictor $\eta_i=x_i'\beta$, with any specified fixed offset added when the model includes one. The replication result applies regardless of link. For an exponential-family GLM with variance $\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)$, the score contribution for $\beta$ is $\{y_i-\mu_i\}(d\mu_i/d\eta_i)x_i/\{a(\phi)V(\mu_i)\}$. Frequency replication multiplies this contribution by $w_i$. The examples make the calculation explicit.

Worked Example: Replicated Bernoulli Observation in a Logistic Model

One row has $y=1$, covariates $x=(1,1)'$, and frequency $w=4$. Consider logistic regression at $\beta=(0,\log 3)'$. The linear predictor is $\eta=x'\beta=\log 3$, so $p=\exp(\eta)/\{1+\exp(\eta)\}=3/4$.

For one Bernoulli observation, the log-likelihood is $y\log p+(1-y)\log(1-p)=\log(3/4)$. The four replicated observations therefore have log-likelihood $4\log(3/4)$. The individual logistic score is $x(y-p)$, so the weighted score is

4x(y-p)=4 \begin{pmatrix}1\\1\end{pmatrix} \left(1-\frac34\right) = \begin{pmatrix}1\\1\end{pmatrix}.

The individual observed negative Hessian is $p(1-p)xx'$, which here equals $\frac{3}{16}\begin{pmatrix}1&1\\1&1\end{pmatrix}$. Multiplying by four gives the replicated observed information $\frac34\begin{pmatrix}1&1\\1&1\end{pmatrix}$. This single covariate pattern contributes a rank-one matrix; replication scales its contribution but does not create a new covariate direction.

Worked Example: Poisson Rows and the Intercept Score

Consider an intercept-only Poisson GLM with log link, so $\mu=\exp(\beta)$. The data have response $y_1=2$ with frequency $w_1=3$, and response $y_2=0$ with frequency $w_2=2$. At $\beta=0$, the mean is $\mu=1$. The Poisson log-likelihood for one response is $y\beta-\exp(\beta)-\log(y!)$. Thus the full log-likelihood is

\ell_w(\beta)=3\{2\beta-e^\beta-\log(2!)\} +2\{0-e^\beta-\log(0!)\} =6\beta-5e^\beta-3\log 2.

At $\beta=0$, this equals $-5-3\log 2$. Differentiating gives $U_w(\beta)=6-5e^\beta$, hence $U_w(0)=1$. The second derivative is $-5e^\beta$, so the observed information at zero is $5$. For each Poisson observation the expected information for the intercept is $\mu$; the replicated expected information is $3(1)+2(1)=5$, in agreement with the observed value here.

Worked Example: Replicated Rows in a Normal Linear Model

Suppose $Y$ is normal with mean $\alpha+\beta x$ and known variance $\sigma^2=4$. There are two rows: $y_1=2,x_1=0,w_1=2$, and $y_2=5,x_2=1,w_2=3$. Evaluate the likelihood, score, and information at $(\alpha,\beta)=(1,2)$. The fitted means are $1$ and $3$, so the residuals are $1$ and $2$.

The normal log-density for one observation is $-\frac12\log(8\pi)-(y-\alpha-\beta x)^2/8$. The weighted log-likelihood is therefore $-\frac52\log(8\pi)-\{2(1^2)+3(2^2)\}/8 =-\frac52\log(8\pi)-\frac74$.

For one observation the score is $x( y-\alpha-\beta x)/4$, where here $x$ denotes the full design vector $(1,x)'$. Summing the two row contributions with their frequencies gives

U_w= \frac14\left[ 2\begin{pmatrix}1\\0\end{pmatrix}(1) +3\begin{pmatrix}1\\1\end{pmatrix}(2) \right] = \begin{pmatrix}2\\3/2\end{pmatrix}.

The information is $X'WX/4$, which can be computed directly from the row design vectors: $\{2(1,0)'(1,0)+3(1,1)'(1,1)\}/4 =\begin{pmatrix}5/4&3/4\\3/4&3/4\end{pmatrix}$. The frequency weights enter once in this sum, because they count replicated observations.

Why the Interpretation Matters

Frequency replication is useful when data are stored compactly: repeated identical records can be represented by one row and a count. The count preserves the likelihood and its derivatives for the observations represented. This is a statement about the data and sampling model, not merely a computational shortcut. In particular, the value of the count determines the amount of information only because it says how many independent observations were made.

A common error is to multiply an individual score by $w_i$ and then conclude that the information contribution is $w_i^2$ times the individual information by squaring that weighted score. That calculation treats the replicated observations as if they were the same random draw repeated $w_i$ times. Under the replication model there are $w_i$ independent observations, and their expected negative Hessians add. Their score variances also add because the independent individual scores have mean zero. Consequently the Fisher information is $w_i\mathcal I_i$, not $w_i^2\mathcal I_i$.

Another error is to assume that any integer-valued weight is a frequency weight. The interpretation must match the data: a count of independent identical records supports replication; an arbitrary importance or precision multiplier does not acquire that interpretation just because it is an integer. Likewise, grouped binomial trial counts describe a response made up of multiple trials, rather than a generic instruction to replicate a single recorded response. The likelihood structure must be identified before its score or information is calculated.

Key takeaway: Under frequency replication, each represented independent observation contributes its own likelihood, score, and expected information. For row $i$, the count $w_i$ multiplies each of those contributions by $w_i$.

Check Your Understanding

Use the replication interpretation in each question.

A row has frequency 6. Write its contribution to the log-likelihood in terms of the individual log-likelihood contribution $\ell_i(\beta)$.
Why does a frequency of 4 multiply the row’s expected information by 4 rather than by 16?
In the Poisson example, what is the total number of represented observations, and what is the score at $\beta=0$?
What independence assumption makes four distinct records representable as a frequency of 4?
Explain why copying one person’s record four times does not justify treating it as four independent observations.

Frequency Weights as Replication Counts

What You'll Learn