Why Candidate Models Must Use Comparable Data

Why Comparability Comes Before Ranking

AIC and BIC turn a model’s maximized likelihood and parameter count into a numerical score. A lower score is often preferred, but that ranking has meaning only when the candidate models are describing the same data on the same basis. If one model is evaluated on more claims than another, or if one likelihood is for individual claim amounts while another is for policy totals, the scores do not answer a like-for-like question.

The key check is not just whether two candidates have the same sample size. They should use the same observational units, the same response definition, and a likelihood calculated under a compatible data and measurement convention. A numerical ranking cannot repair a mismatch in what was observed.

Definition: Candidate models have comparable likelihoods for an information-criterion comparison when their likelihoods describe the same observed units and response, using a consistent probability model and likelihood convention. For ordinary independent observations $x_1,\ldots,x_n$, each candidate’s log likelihood is evaluated on that same set of $n$ observed responses.

For a fitted model $M$, let $\ell_M$ be its maximized log likelihood and $k_M$ its number of free parameters. The usual criteria are

\operatorname{AIC}_M=-2\ell_M+2k_M, \qquad \operatorname{BIC}_M=-2\ell_M+k_M\log n.

Here $n$ is the number of observations represented by the likelihood under the applicable model. For the standard independent-observation setup, it is the number of response observations, not the number of model parameters or the number of records that happen to be available for a different candidate. AIC and BIC can be compared across different model families, but the candidates must still be fitted to comparable data.

Why Different Samples Break a Direct Comparison

For independent observations, the log likelihood is a sum of contributions:

\ell_M(x_1,\ldots,x_n)=\sum_{i=1}^{n}\log f_M(x_i).

If one candidate is evaluated on an additional set of observations, its log likelihood includes additional terms. Those terms change the fit part of AIC and BIC; BIC also changes its penalty through $n$. There is no general adjustment that makes scores from different samples directly comparable: the extra log-likelihood contributions depend on both the observations and the model.

This also explains why equal sample sizes alone are not enough. Two models fitted to different groups of the same size have log likelihoods that sum contributions from different responses. They may be appropriate for separate analyses, but their scores do not establish which model better describes one common dataset.

Justification: Suppose dataset $D_2$ consists of the observations in $D_1$ plus additional observations $E$. For independent data, $\ell_M(D_2)=\ell_M(D_1)+\ell_M(E)$. Thus the AIC difference between two models on $D_2$ includes the difference in their log likelihoods on $E$, as well as the parameter penalties. That extra difference can have either sign. Consequently, an AIC or BIC difference calculated on $D_2$ is not generally the same comparison as one calculated on $D_1$.

Worked Comparisons

Worked Example: Two Severity Models on the Same Claims

A hypothetical insurer fits two continuous severity models to the same 40 individual claim amounts, all recorded in dollars. At their maximum likelihood estimates, Model A has $\ell_A=-120$ and $k_A=2$; Model B has $\ell_B=-116$ and $k_B=3$. Calculate AIC and BIC and identify the preferred model under each criterion.

Method and comparability check. Both likelihoods use the same 40 claim amounts, the same response definition, and the same dollar scale. The comparison is eligible. The criteria use $n=40$.

For Model A,

\operatorname{AIC}_A=-2(-120)+2(2)=240+4=244.

For Model B,

\operatorname{AIC}_B=-2(-116)+2(3)=232+6=238.

Therefore AIC is lower for Model B by $244-238=6$.

For BIC, $\log(40)=3.6889$ to four decimal places. Thus

\begin{aligned} \operatorname{BIC}_A&=240+2\log(40)=240+7.3778=247.3778,\\ \operatorname{BIC}_B&=232+3\log(40)=232+11.0667=243.0667. \end{aligned}

BIC is also lower for Model B, by $247.3778-243.0667=4.3111$. The extra parameter in Model B increases each penalty, but its higher maximized log likelihood more than offsets that increase for both criteria. The calculation avoids the mistake of treating parameter counts alone as the basis for choosing a model.

Worked Example: Different Claim Samples Produce a False Ranking

Two models are fitted to a hypothetical portfolio’s reported claim amounts. Model A uses 60 claims and has $\ell_A=-180$, $k_A=2$. Model B uses only the 48 claims with complete records and has $\ell_B=-137$, $k_B=3$. A data audit shows that Model A’s log likelihood on those same 48 complete claims is $-130$. Compare the tempting full-sample AIC values with a valid comparison on the 48 common claims.

Method and comparability check. The full-sample scores are not directly comparable because they use different claims. They can be calculated to illustrate the trap, but they cannot support a model ranking.

The full-sample AIC values are

\operatorname{AIC}_A=-2(-180)+2(2)=360+4=364, \qquad \operatorname{AIC}_B=-2(-137)+2(3)=274+6=280.

A mistaken reading would say that Model B is preferred by $364-280=84$. The apparent advantage is confounded with the fact that Model A includes 12 additional claims.

Now use the common set of 48 claims. Model A has $\ell_A=-130$ on this set, while Model B has $\ell_B=-137$. The valid AIC calculations are

\operatorname{AIC}_A=-2(-130)+4=264, \qquad \operatorname{AIC}_B=-2(-137)+6=280.

On the common claims, AIC is lower for Model A by $280-264=16$. The result reverses the tempting full-sample ranking. For completeness, $\log(48)=3.8712$. The common-sample BIC values are

\begin{aligned} \operatorname{BIC}_A&=260+2(3.8712)=267.7424,\\ \operatorname{BIC}_B&=274+3(3.8712)=285.6136. \end{aligned}

BIC also prefers Model A on the common claims. The correction is not to adjust the full-sample scores by an arbitrary amount; it is to evaluate both likelihoods on the same 48 claims, if that comparison answers the intended question. This avoids ranking models by scores based on different data.

Worked Example: The Same Losses Expressed in Different Units

Five hypothetical claim amounts are observed. Model P is reported using losses measured in thousands of dollars, with maximized log likelihood $\ell_P=-12$ and $k_P=2$. Model Q is reported using the same losses measured in dollars, with $\ell_Q=-30$ and $k_Q=3$. Put both likelihoods on the dollar scale and compare AIC. Use $\log(1000)=6.9078$.

Method and comparability check. The underlying five claims are the same, but the reported density units differ. A density changes when its response is rescaled, so the raw log likelihoods cannot be compared as if they used the same scale.

Let $Z=X/1000$, where $X$ is the amount in dollars and $Z$ is the amount in thousands. The density transformation is $f_X(x)=f_Z(x/1000)/1000$. For five observations, Model P’s dollar-scale log likelihood is therefore

\ell_P^{(\$)}=\ell_P^{(\text{thousands})}-5\log(1000) =-12-5(6.9078)=-46.5390.

Model Q is already on the dollar scale, so $\ell_Q^{(\$)}=-30$. The AIC values on the common scale are

\operatorname{AIC}_P=-2(-46.5390)+2(2)=93.0780+4=97.0780, \qquad \operatorname{AIC}_Q=-2(-30)+2(3)=60+6=66.

Thus Model Q has the lower AIC by $97.0780-66=31.0780$. If the units were mixed without conversion, Model P would appear to have AIC $-2(-12)+4=28$, incorrectly making it look preferable to Model Q’s 66. The unit conversion changes the numerical log likelihood, not the underlying observations. This example avoids treating a density value as independent of its measurement scale.

Check the Response and Likelihood Definition

A response is more than a column heading. Individual claim severities, policy-level annual totals, and claim counts are different response variables, even if they arise from the same insurance portfolio. A likelihood for one of these quantities cannot be ranked against a likelihood for another merely because both are connected to losses.

The likelihood must also correspond to the same observed-data setup. For example, one candidate might be fitted to every observed record while another is fitted only to a selected subset. Or the candidates might use different definitions of what is recorded as an observation. In such cases, first establish whether the likelihoods can be put on a common basis. Special likelihood constructions for censoring and truncation require their own careful treatment; the comparison principle remains that each candidate must describe the same observed information under a compatible likelihood definition.

The same underlying values expressed in different units can be made comparable by applying the correct transformation to the likelihood. In a common smooth one-to-one response transformation, the density acquires a Jacobian factor. If every candidate is transformed consistently, the resulting common contribution does not alter their ranking. But a candidate reported on one scale cannot be compared by its unadjusted likelihood against another candidate reported on a different scale.

Name the observational unit.
Check whether each likelihood uses the same kind of unit, such as one record per claim or one record per policy.

Specify the response.
Confirm that the models describe the same variable with the same definition, rather than related but different outcomes.

Check the observations and scale.
Verify the records included and the units in which the responses and densities are expressed.

Only then calculate and rank criteria.
Use the common $n$ for BIC and the appropriate parameter count for each model. If the data basis cannot be reconciled, do not report a direct AIC or BIC ranking.

Common Exam Traps

Taking the lowest score without checking the data. A lower AIC from a smaller or different sample does not establish a better model for the full dataset.
Assuming equal $n$ means comparable data. Two disjoint groups can have the same size but different likelihood contributions. Equal sample sizes do not make their criteria comparable.
Mixing response definitions. A model for claim amounts and a model for annual totals do not compete on a common response merely because both concern losses.
Ignoring measurement units. Densities, and therefore log likelihoods, depend on the units of a continuous response. Apply a consistent transformation before comparing them.
Trying to fix an invalid comparison with a parameter penalty. AIC’s $2k$ and BIC’s $k\log n$ penalties account for model complexity; they do not correct for different samples or responses.

Key takeaway: An information criterion ranks candidate models only after their likelihoods are put on a comparable basis. Check the observational units, the exact response, the observations included, and the measurement scale before interpreting which score is lower.

Check Your Understanding

For each item, first decide whether the comparison is valid, then calculate or transform the criterion if requested.

Two models use the same 25 claim amounts. Model A has $\ell=-70$, $k=2$; Model B has $\ell=-67$, $k=3$. Calculate both AIC values and state which is lower. (Answer: AICs are 144 and 140, respectively; Model B is lower.)
Model C uses 30 policy totals and has $\ell=-90$; Model D uses 24 policy totals and has $\ell=-68$. Both have $k=2$. Is a direct AIC ranking valid? (Answer: No; the likelihoods use different policy observations.)
Four losses are expressed in thousands of dollars, with $\ell_Z=-8$. Convert this log likelihood to dollars using $\ell_X=\ell_Z-4\log(1000)$ and $\log(1000)=6.9078$. (Answer: $\ell_X=-35.6312$.)
Two models are fitted to the same 20 observed severities. Model A has $\ell_A=-51$, $k_A=2$; Model B has $\ell_B=-49$, $k_B=3$. Compare AIC. (Answer: AIC values are 106 and 104, respectively; Model B is lower.)
On the same 36 observations, Model E has $\ell_E=-100$, $k_E=2$, and Model F has $\ell_F=-98$, $k_F=4$. Using $\log(36)=3.5835$, calculate BIC and identify the lower value. (Answer: BIC values are 207.1670 and 210.3340, respectively; Model E is lower.)

What You'll Learn