Why weights change model comparisons
Two weighted GLMs can be fitted to the same table and still not support a meaningful comparison. The key question is not simply which model has the larger reported likelihood or smaller deviance. It is whether both fits represent the same response, sampling units, weights and likelihood basis. If any of these change, a numerical difference may reflect the data representation rather than an improvement in the model.
This distinction matters in actuarial work, where a row may represent one observation, several identical observations, or a group of binomial trials. Those cases can lead to different weight meanings. A likelihood-ratio test additionally requires nested models and a valid likelihood-based sampling interpretation. AIC can compare non-nested models, but still requires comparable likelihoods for the same response basis.
Conditions for likelihood-based comparisons
For a likelihood-ratio comparison, let \(M_0\) be a reduced model and \(M_1\) a larger model. They must be nested: every possible fitted distribution under \(M_0\) must also be available under \(M_1\). In addition, both models must use the same response observations, response encoding, weights and likelihood basis. The likelihood must describe the sampling process, rather than merely serve as a convenient weighted objective.
If \(M_0\) and \(M_1\) satisfy these conditions, the likelihood-ratio statistic is
where \(\ell(M)\) is the maximized log likelihood under model \(M\). Under the null model and standard regularity conditions, \(G^2\) is approximately distributed as a chi-square random variable with degrees of freedom equal to the difference in the number of freely estimated parameters. This approximation can be unreliable with small effective samples, boundary parameters or other violations of regularity conditions.
AIC is defined by \( \mathrm{AIC}=-2\ell+2k\), where \(k\) is the number of estimated parameters in the likelihood used. AIC does not require nested models, but it does require likelihoods on a comparable basis. In practice, compare models fitted to the same response representation and effective data, using the same likelihood convention and weights. An additive likelihood constant shared by every candidate on that basis does not affect their AIC ordering; a constant that changes across representations does.
Why the likelihood-ratio formula works
The likelihood ratio compares how well each model can explain the same observed data. The likelihood under \(M_1\) cannot be smaller than under \(M_0\) when \(M_1\) contains \(M_0\), because \(M_1\) can always reproduce the reduced model's fitted distribution. Taking a log turns the likelihood ratio into a difference of log likelihoods, and multiplying by two gives the standard statistic:
The chi-square approximation is a further result, not part of this algebraic identity. It applies asymptotically under a valid likelihood, nested models and regularity conditions. A weighted objective alone does not guarantee those conditions.
Worked Example: Comparing Two Grouped-Binomial Models
In a hypothetical service study, a response is observed for 4 independent cases in each of two risk tiers. Tier A has 2 responses and 2 nonresponses; tier B has 3 responses and 1 nonresponse. Compare an intercept-only binomial model \(M_0\) with a model \(M_1\) having a separate response probability for each tier. Use the grouped-binomial likelihood, including its binomial coefficients.
Method and conditions. The models use the same two groups and trial counts. The intercept-only model is nested in the tier model, which adds one parameter. The response is represented as successes and failures out of the same group-specific trial counts, so a likelihood-ratio comparison is appropriate under the usual binomial assumptions.
Under \(M_0\), the fitted probability is the pooled proportion: \( \hat p=(2+3)/(4+4)=5/8=0.625\). Under \(M_1\), the fitted probabilities are \( \hat p_A=2/4=0.5\) and \( \hat p_B=3/4=0.75\).
For a group with \(y\) responses from \(n\) trials, the log likelihood contribution is \( \log {n\choose y}+y\log p+(n-y)\log(1-p)\). For \(M_0\), the two binomial coefficient terms sum to \( \log {4\choose2}+\log {4\choose3}=\log6+\log4=3.178054\). The probability terms are \(5\log(0.625)+3\log(0.375)=-5.292506\). Thus \( \ell(M_0)=3.178054-5.292506=-2.114452\).
For \(M_1\), the same coefficient terms apply. The probability terms are \(2\log(0.5)+2\log(0.5)+3\log(0.75)+\log(0.25)=-5.021929\). Thus \( \ell(M_1)=3.178054-5.021929=-1.843875\). The log-likelihood improvement is \(-1.843875-(-2.114452)=0.270577\), giving \(G^2=2(0.270577)=0.541153\). The degrees of freedom are \(2-1=1\).
For AIC, \(M_0\) has one fitted parameter and \(M_1\) has two: \( \mathrm{AIC}(M_0)=-2(-2.114452)+2(1)=6.228904\); \( \mathrm{AIC}(M_1)=-2(-1.843875)+2(2)=7.687751\). The intercept-only model has the lower AIC in this example, despite the larger model fitting at least as well. This is a small dataset, so the result should not be treated as a general claim about the population.
Check and common mistake. The larger model's log likelihood is higher, as nesting requires, and the likelihood-ratio statistic is nonnegative. Do not confuse better in-sample likelihood with lower AIC: AIC includes a parameter penalty. The likelihood-ratio statistic is \(0.5412\) and the AIC difference, larger minus smaller, is \(7.687751-6.228904=1.458847\).
Frequency weights and the effective data basis
If a row represents \(w_i\) identical independent observations with response \(y_i\) and fitted mean \(\mu_i\), a frequency-weighted likelihood contribution is the single-observation log likelihood multiplied by \(w_i\). Consequently, multiplying every frequency by a constant represents more replications, not merely a different way to label the same evidence. The fitted coefficients can remain unchanged while the likelihood improvement and deviance difference scale with the replication count.
Worked Example: What Happens When Frequency Weights Double?
Suppose a hypothetical Poisson dataset has two covariate groups. One observed count is \(y_1=1\), and the other is \(y_2=3\). Compare an intercept-only model with a model that estimates a separate mean for each group. First use frequency weights of 1 for both rows; then consider weights of 2 for both rows.
Method. Under the Poisson likelihood, a row with count \(y\) and mean \(\mu\) contributes \(y\log\mu-\mu-\log(y!)\). The intercept-only estimate is the overall mean, \(\hat\mu=(1+3)/2=2\). The group model has fitted means 1 and 3.
For the group model, the log likelihood is \([1\log1-1-\log(1!)]+[3\log3-3-\log(3!)]\). The first contribution is \(-1\). The second is \(3(1.098612)-3-1.791759=-1.495923\). Therefore \( \ell(M_1)=-2.495923\).
For the intercept-only model, the contributions are \(1\log2-2-\log(1!)=0.693147-2=-1.306853\) and \(3\log2-2-\log(3!)=2.079442-2-1.791759=-1.712318\). Therefore \( \ell(M_0)=-3.019171\). The improvement is \(-2.495923-(-3.019171)=0.523248\), and \(G^2=2(0.523248)=1.046496\).
With frequency weight 2 on both rows, each log-likelihood contribution is counted twice. The group model log likelihood becomes \(-4.991845\), the intercept-only log likelihood becomes \(-6.038341\), and the improvement becomes \(1.046496\). Thus the new likelihood-ratio statistic is \(2.092992\). The fitted means have not changed, but the effective replicated data have changed.
Check and common mistake. The weighted statistic is exactly twice the unweighted statistic, as the frequency-weight definition implies. Do not multiply frequencies and then describe the fits as based on the original effective sample. If weights instead encode a different precision or sampling design, this replication interpretation may not be valid.
Grouped responses, quasi-likelihood and limits
For grouped binomial data, the response can be represented as successes and failures, or as a success proportion with the number of trials supplied as the binomial prior weight. These are equivalent only when the software call and likelihood interpret the response and weights consistently. A trial count is not an arbitrary multiplier to add on top of a successes-and-failures response.
Response encoding can also affect reported likelihood values. The grouped-binomial probability of observing a count includes a binomial coefficient. The likelihood of one particular expanded sequence of binary outcomes does not include that coefficient. For a fixed group, this difference is constant across models, so likelihood-ratio differences and AIC differences within either encoding agree. The raw log likelihoods and AIC values across encodings need not agree.
Quasi-likelihood requires still more caution. A quasi-binomial model specifies a mean-variance relationship with a dispersion parameter, but does not specify a full probability distribution and likelihood. Its deviance can be useful as a fit summary, and scaled deviance may be used under a stated inferential procedure. It does not, by itself, justify an ordinary likelihood-ratio test or an ordinary AIC calculation.
Worked Example: A Quasi-Binomial Deviance Is Not a Likelihood Ratio
A hypothetical grouped response study has 5 trials in each of two risk groups. Group C has 2 responses and group D has 4. Compare an intercept-only mean with group-specific means. Suppose the analysis uses a quasi-binomial variance specification with dispersion \( \phi=2\). Calculate the binomial deviance difference and its simple dispersion-scaled value, then state what the calculation does and does not establish.
Method. The pooled response proportion under the intercept-only mean is \(6/10=0.6\). Under the group model, the fitted proportions are \(2/5=0.4\) and \(4/5=0.8\). The binomial deviance contribution for observed successes \(y\), failures \(n-y\), and fitted probability \(p\) is \(2[y\log\{y/(np)\}+(n-y)\log\{(n-y)/(n(1-p))\}]\).
For group C under the pooled fit, expected successes and failures are 3 and 2. The half-deviance contribution is \(2\log(2/3)+3\log(3/2)=-0.810930+1.216395=0.405465\). For group D it is \(4\log(4/3)+\log(1/2)=1.150728-0.693147=0.457581\). The total deviance difference is \(2(0.405465+0.457581)=1.726092\). Dividing by the stated dispersion gives \(1.726092/2=0.863046\).
Check and common mistake. The group-specific model is saturated for these two group proportions, so its residual deviance is zero; the calculated difference is the pooled model's deviance relative to that fit. The arithmetic checks: the two half-deviance contributions sum to \(0.863046\), and doubling gives \(1.726092\). The scaled value \(0.863046\) is a deviance-based summary under the stated dispersion, not an ordinary likelihood-ratio statistic. Because a quasi-likelihood does not supply a full likelihood, do not label it a standard chi-square likelihood-ratio test or calculate ordinary AIC from it.
A comparison checklist
Check that the models use the same observations and response encoding, such as the same grouped-binomial trial counts.
Establish whether they represent replication, binomial trials, precision, or another design feature. Do not assume these meanings are interchangeable.
Use a likelihood-ratio test for nested models with a valid likelihood and suitable regularity conditions. Use AIC only with comparable likelihoods and a consistent likelihood convention.
State the data basis, weight interpretation, statistic and relevant assumptions. If the fit uses quasi-likelihood or a design-based weighting scheme, do not imply ordinary likelihood inference without justification.
When checking weighted GLM comparisons in R, verify that the models were fitted to the same effective rows and that the response and weights have the intended meaning. A difference in reported deviance or AIC is useful only after that verification. This discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the relevant model-comparison condition or calculate the requested statistic.
- A grouped-binomial dataset has 4 responses out of 4 trials in group A and 2 responses out of 4 in group B. An intercept-only model is compared with a model having a separate probability by group. Are the models nested, and how many degrees of freedom separate them? (Answer: Yes; 1 degree of freedom.)
- Two Poisson rows have counts 2 and 4, each with frequency weight 1. Compare an intercept-only mean of 3 with separate group means of 2 and 4. Calculate the likelihood-ratio deviance difference. (Answer: \(2[2\log(2/3)+4\log(4/3)]=0.6796\).)
- For the Poisson comparison in question 2, both frequency weights are changed from 1 to 3. What is the new likelihood-ratio statistic if these are true replication counts? (Answer: \(3(0.6796)=2.0388\), using the unrounded statistic before rounding.)
- A grouped-binomial dataset has 1 response out of 3 trials in group A and 3 responses out of 5 in group B. Compare the pooled probability \(4/8=0.5\) with the group-specific probabilities. What is the binomial deviance difference? (Answer: \(2[\,\log(2/3)+2\log(4/3)+3\log(1.2)+2\log(0.8)\,]=0.5412\).)
- A quasi-binomial comparison has a deviance difference of \(0.5412\) and a stated dispersion of \(1.5\). What is the dispersion-scaled deviance summary, and does it alone justify an ordinary likelihood-ratio test? (Answer: \(0.5412/1.5=0.3608\); no, a quasi-likelihood does not provide an ordinary full-likelihood ratio.)