Frequency Weights and Replicated Observations

When Does a Weight Mean “Repeat This Record”?

A data table may contain one row for a covariate-response combination that occurred many times. If the row records the same predictor values and the same response for each occurrence, an integer frequency weight can compact those identical records. The compressed table is then a shorter representation of the same observed data, not a new kind of observation.

The essential condition is exact repetition of the entire modeled record: both the response and every predictor value must be identical across the represented observations. A weight of 4 does not mean that four different responses can be replaced by their average. Nor does an integer weight automatically have frequency meaning; its meaning depends on how the data were constructed.

Definition: A frequency-weighted record with weight \(m_j\), a nonnegative integer, represents \(m_j\) independent observations having the same response \(y_j\) and the same modeled predictor vector \(x_j\). Under the same GLM and likelihood assumptions, its log-likelihood contribution is \(m_j\) times the contribution of one such observation.

This is a specific interpretation of weights, not a general rule that all weighted GLMs equal fits to replicated records. In particular, prior weights may encode other information. The earlier material on prior weights distinguishes their general role in a GLM; here the question is whether the data truly are identical repetitions.

Why the Likelihood Contributions Multiply

Suppose the model gives one observation with predictor vector \(x_j\) and response \(y_j\) a probability mass or density \(f(y_j\mid x_j;\beta)\), where \(\beta\) is the coefficient vector. If that exact record appears \(m_j\) times and observations are conditionally independent, the likelihood contribution is the product of \(m_j\) identical terms.

\prod_{r=1}^{m_j} f(y_j\mid x_j;\beta) = \{f(y_j\mid x_j;\beta)\}^{m_j}.

Taking logarithms converts this product into a sum. Thus, for compact records indexed by \(j=1,\ldots,J\), the expanded-data log-likelihood and the frequency-weighted log-likelihood are both

\ell(\beta) = \sum_{j=1}^{J}m_j\log f(y_j\mid x_j;\beta).

Because the objective function is the same, its maximizing coefficient estimates are the same, subject to the same model, link, offset, and estimation conditions. The same argument applies to likelihood-based deviance contributions: each record's contribution is counted \(m_j\) times. This equality concerns a true expansion into identical records. It does not say that every summary printed for a compact data frame must match the summary for the expanded data frame.

Worked Example: A Poisson Intercept Fit With Repeated Records

A hypothetical portfolio extract has two compact rows. One row has claim count \(y=1\) and frequency 2; the other has claim count \(y=5\) and frequency 1. Fit an intercept-only Poisson GLM, and compare the frequency-weighted data with the three explicitly listed counts.

Method. For independent Poisson responses with a common mean \(\mu\), the log-likelihood for expanded observations is \(\sum_r[y_r\log(\mu)-\mu-\log(y_r!)]\). The frequency-weighted form sums each row's contribution multiplied by its frequency.

Estimate the mean. The expanded responses are \(1,1,5\), so their total is \(1+1+5=7\) and their number is \(2+1=3\). The intercept-only Poisson maximum-likelihood estimate is the sample mean:

\hat{\mu} = \frac{2(1)+1(5)}{2+1} = \frac{7}{3} \approx 2.3333.

The compact fit gives the same result because its weighted total is 7 and its total frequency is 3. In R, the relevant comparison is a Poisson GLM with an intercept and weights \(2,1\), versus a Poisson GLM on responses \(1,1,5\) with no weights.

Check the log-likelihood. For the expanded records, the two responses equal to 1 contribute \(2[\log(7/3)-7/3]\). The response equal to 5 contributes \(5\log(7/3)-7/3-\log(120)\). Therefore,

\ell = 7\log(7/3)-7-\log(120) \approx -5.8564.

The compact weighted likelihood gives the same expression: the \(y=1\) contribution is multiplied by 2, and the \(y=5\) contribution by 1. This verifies equality of the likelihood as well as the estimated mean.

Check the deviance. For Poisson responses, with the convention that \(0\log(0/\hat{\mu})=0\), the deviance is

D = 2\sum_j m_j \left[ y_j\log\left(\frac{y_j}{\hat{\mu}}\right) -(y_j-\hat{\mu}) \right].

For \(y=1\), the bracket is \(\log(3/7)-(1-7/3)\approx0.4860\); applying frequency 2 gives \(0.9721\). For \(y=5\), the bracket is \(5\log(15/7)-(5-7/3)\approx1.1440\); frequency 1 leaves it at \(1.1440\). The sum inside the outer factor of 2 is \(0.9721+1.1440=2.1161\), so \(D\approx4.2322\). The expanded data have the same two contributions, with the \(y=1\) contribution occurring twice.

Check. The fitted mean is the total number of claims divided by the total number of observations, \(7/3\), and the weighted deviance counts the two \(y=1\) records separately. The common mistake avoided here is treating the compact table as though it contained only two observations.

Equivalence Has Specific Conditions

For exact replication, a compact row and each represented expanded row must use the same response family and model specification. This includes the same predictor values and transformations, factor coding, offset, and any other model terms. If an offset differs among the purported repetitions, they are not identical modeled records. The responses must also be represented in the way the GLM family expects; a frequency weight does not change the response definition.

Integer frequency weights can preserve coefficient estimates and likelihood-based quantities such as deviance when the records are genuine repetitions. However, users should inspect the model output rather than assume that every degree of freedom or summary statistic is automatically based on the total frequency. In R, the GLM fitting machinery uses the supplied weights in fitting and deviance calculations, while residual degrees of freedom are based on the input rows with positive weights, less the fitted rank. Consequently, compressing repeated rows can produce different reported residual degrees of freedom from an explicitly expanded data set. This matters especially when interpreting dispersion estimates and standard errors for models where dispersion is estimated.

For a Poisson GLM with dispersion fixed at 1, the coefficient estimates, fitted means, log-likelihood, and deviance are the important direct comparisons in the preceding example. Do not infer from their equality that all inferential output is necessarily identical. When a particular diagnostic depends on the number of observations or residual degrees of freedom, verify how the compact representation affects it.

Worked Example: A Gaussian Regression and Its Expanded Data

A hypothetical analyst models a continuous response \(Y\) from a numeric predictor \(x\), using a Gaussian GLM with identity link and constant variance. The compact data are \((x,y)=(0,1)\) with frequency 2, \((1,2)\) with frequency 1, and \((2,2)\) with frequency 2. Find the regression coefficients and weighted residual sum of squares, then compare with the expanded data.

Method. A Gaussian GLM with identity link has fitted mean \(\beta_0+\beta_1x\). Maximizing its likelihood for a common variance gives the ordinary least-squares coefficient estimates; frequency weighting repeats each squared residual according to its frequency.

The total frequency is \(n=2+1+2=5\). The weighted sums are \(\sum m_jx_j=0(2)+1(1)+2(2)=5\), \(\sum m_jy_j=1(2)+2(1)+2(2)=8\), \(\sum m_jx_j^2=0+1+8=9\), and \(\sum m_jx_jy_j=0+2+8=10\). Thus \(\bar{x}=5/5=1\) and \(\bar{y}=8/5=1.6\).

\hat{\beta}_1 = \frac{\sum m_jx_jy_j-n\bar{x}\bar{y}} {\sum m_jx_j^2-n\bar{x}^{\,2}} = \frac{10-5(1)(1.6)}{9-5(1)^2} = \frac{2}{4} = 0.5, \qquad \hat{\beta}_0 = \bar{y}-\hat{\beta}_1\bar{x} = 1.6-0.5 = 1.1.

The fitted means at \(x=0,1,2\) are respectively \(1.1,1.6,2.1\). The weighted residual sum of squares is

$$ 2(1-1.1)^2+(2-1.6)^2+2(2-2.1)^2 = 2(0.01)+0.16+2(0.01) = 0.20. $$

The expanded data contain \((0,1),(0,1),(1,2),(2,2),(2,2)\). Their \(x\)- and \(y\)-totals, cross-product, fitted coefficients, and squared-error sum are exactly the same: each compact-row calculation is simply repeated the specified number of times. The weighted coefficient fit and deviance therefore match the expanded fit. But the compact data have three input rows and the expanded data have five; in R, the residual degrees of freedom can accordingly differ. Do not use the compact fit's residual degrees of freedom as though the table had been expanded.

Check. The fitted line predicts \(1.1,1.6,2.1\), and the five expanded squared residuals sum to \(0.01+0.01+0.16+0.01+0.01=0.20\), agreeing with the weighted calculation. The common mistake avoided is calculating the coefficients from only three unweighted rows.

Averaging Is Not Replication

A tempting compression is to replace several different outcomes at the same predictor values by their average, then use the number of original observations as a weight. That can preserve the weighted mean for some coefficient calculations, but it generally does not preserve the likelihood, deviance, or residual variation. The rows do not represent repeated identical responses.

Worked Example: The Trap of Replacing Different Counts by Their Average

Consider two Poisson responses, \(y=1\) and \(y=5\), each observed once at the same predictor values. Compare their deviance at a fixed fitted mean of 3 with the deviance obtained by replacing them with their average and assigning frequency 2.

Method. Apply the Poisson deviance formula to each original response, then separately to the purported compressed record. This comparison tests whether the response records are identical, not merely whether their average is preserved.

For the original data, the \(y=1\) bracket is \(\log(1/3)-(1-3)=\log(1/3)+2\approx0.9014\). The \(y=5\) bracket is \(5\log(5/3)-(5-3)\approx0.5541\). Therefore,

D_{\text{original}} = 2\left[ \log(1/3)+2 + 5\log(5/3)-2 \right] \approx 2(0.9014+0.5541) = 2.9110.

The average response is \((1+5)/2=3\). A single response of 3 has zero Poisson deviance at fitted mean 3, so assigning it frequency 2 still gives \(D_{\text{average}}=2(0)=0\). The difference, \(2.9110\) versus 0, shows that averaging has discarded the variation in the original outcomes. This is not an equivalent frequency-weighted representation.

Check. The original deviance is approximately 2.9110, not 1.9253; the averaged-record deviance is 0. The common mistake is to treat a weighted average as though it were a set of identical observations.

Practical Checks in R and Common Traps

For a genuine frequency table, the practical check is to fit both representations using the same formula and family. In R, a weighted fit can use a frequency column in the weights argument; the expanded comparison repeats each response and all of its predictors according to that frequency and fits without those weights. Compare coefficients, fitted values at corresponding records, and deviance. For likelihood-based comparisons, also confirm that the response family and likelihood conventions are the same.

Before interpreting the result, verify the following:

Confirm record identity. Every predictor, response, offset, and other modeled quantity must be the same across the observations represented by one row.
Check the weight's meaning. A frequency of 3 represents three records only when that is how the data were generated or summarized. A number that happens to be an integer is not sufficient evidence.
Do not average unlike responses. Identical predictors do not make distinct outcomes identical; averaging can remove deviance and variation.
Separate fit from summaries. Coefficient and deviance equivalence does not guarantee that compact-data residual degrees of freedom or every inferential summary equals the expanded-data output.
Do not substitute exposure or trial information. A frequency weight is not automatically an offset or a count of binomial trials. Those quantities represent different model information.

The central test is simple: could the compact table be expanded into exactly the same response-and-predictor rows without inventing or changing any value? If so, frequency weighting reproduces the repeated-record likelihood. If not, treat the weight according to its actual statistical meaning rather than assuming replication.

Check Your Understanding

For each item, decide what the frequency representation preserves and identify any qualification required.

A Poisson intercept-only data set has counts 2 and 4, each recorded once. What is the fitted mean? If the two rows are instead each assigned frequency 3, what is the fitted mean? (Answer: 3; 3.)
A Gaussian regression contains one record \((x,y)=(3,7)\) with frequency 4. What records appear in an explicit expansion, and what happens to the weighted squared-error contribution for that point? (Answer: Four copies of \((3,7)\); its squared residual is counted four times.)
Two Poisson outcomes at the same predictors are 0 and 6. An analyst replaces them by response 3 with frequency 2. Is this exact frequency replication? (Answer: No; the original responses differ, so averaging does not reproduce their likelihood or deviance.)
A compact R GLM has four rows with positive integer frequencies totaling 20. Does that fact alone guarantee that every residual-degree-of-freedom summary matches a fit on 20 expanded rows? (Answer: No; R's residual degrees of freedom can be based on the four positive-weight input rows, less the fitted rank.)
A row has frequency 5, but its five underlying observations share the same response and predictors while having different offsets. Can this one row be an exact replicated record for the stated GLM? (Answer: No; the modeled records differ because their offsets differ.)

What You'll Learn