Comparing Likelihoods on a Common Data Set

Why a Common Data Set Matters

A likelihood summarizes how compatible a specified model is with the observations it is asked to explain. Comparing two likelihoods can help assess competing models, but the numerical comparison only has a clear interpretation when both likelihoods refer to the same observed responses, defined in the same way. If one model is evaluated on more records, different outcomes, or a different response scale, the values can reflect those changes rather than a difference in model fit.

This is a practical issue when fitting GLMs: two calls may silently use different rows because of missing predictors, different outcome coding, or a changed data filter. Before interpreting likelihoods, verify the response, the observation units, the records used, and the likelihood convention. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For independent response observations $Y_1,\ldots,Y_n$, a model with parameters $\theta$ assigns a probability mass or density $f(y_j\mid\theta)$ to each observed response $y_j$. Its likelihood for the observed response vector is $$ L(\theta;y_1,\ldots,y_n)=\prod_{j=1}^{n} f(y_j\mid\theta), $$ and its log-likelihood is $$ \ell(\theta;y_1,\ldots,y_n)=\sum_{j=1}^{n}\log f(y_j\mid\theta). $$ These expressions assume that the observations are independent conditional on the model inputs and that the stated probability model is appropriate. The likelihood is a function of $\theta$ with the observed responses held fixed; it is not the probability that the model is true.

What Must Be Held Fixed

When comparing two fitted models by their likelihoods, both must describe the same response observations. This means more than using the same data-frame name. The following must agree:

Response records: the same observational units must contribute to both likelihoods, including units with zero outcomes.
Response definition: the outcome must represent the same event or quantity in both models. A claim count is not the same response as an indicator for whether any claim occurred.
Response representation: observations should be expressed at compatible levels, such as individual binary outcomes or grouped binomial counts, with likelihood constants handled consistently.
Likelihood basis: weights, exposure treatment, censoring rules, and other features that define each observation's contribution must be the same or explicitly accounted for.
Probability model: each likelihood must be a genuine likelihood for the stated response distribution. A quasi-likelihood does not specify a full probability distribution and cannot be treated as an ordinary likelihood value.

The models need not have the same predictors or fitted means; those differences are often the reason to compare them. But the observations being scored must be common. If the response values or sample space changes, a numerical likelihood difference does not isolate a change in fit.

Why the Log-Likelihood Is a Sum

Suppose two models, $M_1$ and $M_2$, assign probabilities or densities to the same independent observations $y_1,\ldots,y_n$. Conditional independence gives a product of contributions for each model. Taking logarithms converts that product to a sum:

\ell_k=\log\left\{\prod_{j=1}^{n} f_k(y_j)\right\} =\sum_{j=1}^{n}\log f_k(y_j),\qquad k\in\{1,2\}.

Therefore, each observation contributes to both models' scores. The difference $\ell_1-\ell_2$ compares the two models' assigned probability or density for the same response vector. If the first sum contains three observations and the second contains only two, the difference also reflects an unequal set of contributions. It is no longer a like-for-like comparison.

This also explains a useful nuance about grouped data. Re-expressing individual Bernoulli outcomes as binomial counts can introduce combinatorial factors, such as $\binom{n}{y}$. If the same grouping is used for both models, these factors do not depend on the model parameters and cancel in a likelihood ratio. But likelihood values on the individual and grouped conventions have different numerical scales. Their constants must not be mixed or silently dropped when reporting a direct likelihood comparison.

Worked Example: Comparing Two Bernoulli Models on the Same Records

A hypothetical portfolio study records whether each of three policies had a weather-related claim during one year. The outcomes are $y=(1,0,1)$, where 1 means at least one claim. Assume independent Bernoulli responses. Model A assigns claim probability $p=0.50$ to every policy; Model B assigns $p=0.75$ to every policy. Compare their likelihoods for these observations.

Method. For independent Bernoulli outcomes, an observation $y_j$ contributes $p^{y_j}(1-p)^{1-y_j}$. The response vector and its three policy records are fixed for both models.

For Model A, the probabilities of the observed outcomes are $0.50$, $0.50$, and $0.50$. Thus $$ L_A=(0.50)(0.50)(0.50)=0.125. $$ Its log-likelihood is $\ell_A=\log(0.125)=-2.0794$, to four decimal places.

For Model B, the first and third outcomes are successes and the second is a failure: $$ L_B=(0.75)(0.25)(0.75)=0.140625. $$ Its log-likelihood is $\ell_B=\log(0.140625)=-1.9617$, to four decimal places. Since $0.140625>0.125$, Model B assigns the observed vector a higher likelihood.

Check. There are two successes and one failure, so $L_B=(0.75)^2(0.25)=0.140625$, agreeing with the product above. The common mistake avoided is comparing models evaluated on different policies; here all three outcomes enter each likelihood.

Changes That Make a Direct Comparison Invalid

A changed data set can invalidate a comparison even when the model family and response name appear unchanged. For example, software may omit a policy with a missing predictor from one fitted model but not from the other. Dropping zero outcomes is another consequential change: those observations contribute probability information and are not interchangeable with absent records.

Changing how the response is recorded is also important. A count of claims includes the distinction between one claim and several claims. A binary indicator for any claim discards that information and has a different response distribution. Comparing the resulting likelihood numbers does not tell us which model better describes one common outcome.

A response transformation or change of units needs particular care. For a continuous response, a density is expressed per unit of the response scale. For example, a density per dollar and a density per thousand dollars differ by a scale factor. A properly transformed likelihood accounts for the corresponding density change, but raw density values calculated under different units should not be compared as if they were on the same scale.

Worked Example: The Effect of Dropping a Response Record

For three hypothetical policies, the binary claim outcomes are $y=(1,0,0)$. Assume independent Bernoulli responses. Model A assigns the same claim probability $p=0.50$ to every policy. Model B assigns $p=0.80$. A data-processing difference means Model B is reported using only the first, positive-outcome record. Show why the reported values cannot be directly compared, then calculate a valid comparison on common records.

Method. First calculate each model's likelihood using the same three outcomes. Then show what happens if Model B's likelihood is calculated on only one outcome.

On all three records, Model A has $$ L_A=(0.50)(0.50)(0.50)=0.125. $$ Model B on all three records has one success and two failures, so $$ L_B=(0.80)(0.20)(0.20)=0.032. $$ On this common data set, Model A assigns the observed outcomes the higher likelihood: $0.125>0.032$.

If Model B is instead evaluated only on the first record, its likelihood is $0.80$. Comparing $0.80$ with Model A's $0.125$ is invalid: the first number describes one observation and the second describes three. Recalculate Model A on that same one-record subset: its likelihood is $0.50$. On that subset, Model B's likelihood is higher, since $0.80>0.50$.

Check. The conclusion changes with the records being scored, which is precisely why the subset must be stated. The common mistake avoided is treating a larger likelihood based on fewer observations as evidence of better fit on the full data set.

Grouped Responses, Weights, and Exposure

Worked Example: Individual and Grouped Binomial Likelihoods

A hypothetical survey records two groups. Group 1 has 3 successes among 4 independent Bernoulli trials, and Group 2 has 1 success among 3 trials. Compare a model with success probability $p=0.50$ in both groups to a model with $p_1=0.60$ and $p_2=0.40$. Assume the trials are independent within and between groups.

Method. Calculate the likelihood from the individual trial outcomes, then calculate it from grouped binomial counts. Keep the combinatorial factors consistent across the models.

At the individual-trial level, the first model has four successes and three failures overall. Its likelihood is $$ L_0=(0.50)^4(0.50)^3=(0.50)^7=0.0078125. $$ For the second model, Group 1 contributes $(0.60)^3(0.40)$, and Group 2 contributes $(0.40)(0.60)^2$. Thus $$ L_1=(0.60)^3(0.40)(0.40)(0.60)^2 =(0.60)^5(0.40)^2=0.0124416. $$ On this individual-outcome convention, the second model has the larger likelihood.

For grouped binomial counts, the first model's likelihood includes the number of ways to arrange the successes within each group: $$ L_0^{(g)}=\binom{4}{3}\binom{3}{1}(0.50)^7 =(4)(3)(0.0078125)=0.09375. $$ The second model's grouped likelihood is $$ L_1^{(g)}=\binom{4}{3}\binom{3}{1}(0.60)^5(0.40)^2 =(12)(0.0124416)=0.1492992. $$ The model ranking is unchanged because both grouped likelihoods have the same factor 12. But $0.1492992$ must not be compared directly with the individual-outcome value $0.0078125$: they use different likelihood representations.

Check. The grouped-to-individual multiplier is 12 for each model, so it cancels from their ratio. The common mistake avoided is interpreting a representation-dependent constant as a change in relative fit.

Weights and exposure require the same discipline. A frequency weight represents repeated observations; changing it changes how much that response contributes to the likelihood. Other weight conventions may have a different interpretation. Similarly, an offset such as log exposure changes the mean structure, but the exposure values and their treatment must be matched if the models are intended to explain the same rate observations. Do not assume that two fits used the same likelihood contributions merely because they share a response column.

For a reliable GLM comparison, inspect the actual model frames and confirm that the response values, row identities, weights, and relevant exposure information agree. Also confirm that both fits use a full probability model. For example, quasi-Poisson specifies a mean-variance relationship but not a full probability mass function; it therefore does not supply an ordinary likelihood value to compare in the same way as a Poisson model.

A Practical Comparison Checklist

Identify the response precisely.
Write down the outcome meaning, coding, observational unit, and scale. Distinguish, for instance, a count from an indicator of any event.

Match the contributing records.
Check row identities after missing-value handling and filters. Verify that zero outcomes have not been removed from one fit.

Check the likelihood convention.
Confirm compatible treatment of grouped data, weights, exposures, and constants. Make sure each fit supplies a full likelihood rather than only a quasi-likelihood.

Interpret only after these checks.
Once the response and data basis are common, a likelihood difference reflects how the models assign probability or density to those shared observations.

Key takeaway: A likelihood is tied to a particular response vector and a particular probability model. For a direct comparison, use the same response observations and definition, and keep the data representation and likelihood convention compatible. If those conditions fail, fix the comparison basis before interpreting the numbers.

Check Your Understanding

For each problem, decide what can be compared and calculate any requested likelihoods using the stated model.

Independent Bernoulli outcomes are $1,1,0$. Model A uses $p=0.60$ for all three observations, and Model B uses $p=0.50$. Calculate both likelihoods and identify the larger one. (Answer: $L_A=0.1440$, $L_B=0.1250$; Model A is larger.)
Independent Poisson responses are $y=(0,2)$. Model A specifies means $(1,1)$, while Model B specifies means $(1,2)$. Using $P(Y=y)=e^{-\mu}\mu^y/y!$, calculate both log-likelihoods to four decimals and identify the larger one. (Answer: $\ell_A=-2-\log 2=-2.6931$, $\ell_B=\log 2-3=-2.3069$; Model B is larger.)
A binary-outcome model uses 80 policies, but a second model silently omits 12 policies with zero outcomes because one predictor is missing. Can their raw likelihoods be directly compared? State the required correction. (Answer: No; refit or evaluate both models on the same policies, or use a separately justified approach that scores both on a common response set.)
Two analysts use the same grouped binomial counts and the same models. One includes binomial coefficients in each likelihood and the other omits them. Can they compare the two models' likelihood ratios if the convention is applied consistently to both models? Explain briefly. (Answer: Yes, for these same grouped counts the shared binomial-coefficient factors cancel from the ratio; the reported absolute likelihoods use different conventions and should not be mixed.)
One model treats each policy's number of claims as a Poisson response. Another treats whether the policy had any claim as a Bernoulli response. Is a direct comparison of their raw likelihood values valid? (Answer: No; the response definitions and probability models differ, so the likelihoods are not for the same response vector.)

What You'll Learn