A Reproducible Information-Criterion Selection Workflow

Why a Reproducible Workflow Matters

Information criteria are useful only when the comparison is like-for-like. A ranking can change if the candidate models were fitted to different observations, if one likelihood includes a constant that another omits, or if the parameter count is inconsistent. A reproducible workflow makes those choices visible before they affect the conclusion.

This tutorial sets out a practical sequence for comparing candidate models. The information criteria are part of that sequence, not a substitute for defining the data, checking fit, or explaining the actuarial purpose. The workflow is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Definition: For a candidate model fitted by maximum likelihood, let $\ell(\hat{\theta})$ be its maximized log-likelihood, let $k$ be the number of freely estimated parameters, and let $n$ be the number of observations under the stated BIC convention. The Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2k,\qquad \mathrm{BIC}=-2\ell(\hat{\theta})+k\ln(n). $$ For both criteria, a smaller value is preferred among the models being compared. This ranking is meaningful only when the models use a comparable likelihood for the same data.

The likelihood must represent the data actually observed. For example, a likelihood for observations subject to censoring must account for the censoring information; it cannot be compared casually with a likelihood treating every value as fully observed. The parameter count includes all freely estimated model parameters, not fixed constants or quantities specified in advance. When the BIC sample-size convention is not obvious, state and justify it rather than silently choosing a count.

A Repeatable Selection Workflow

A selection record should allow another analyst to reconstruct both the calculation and the reasoning. Use the following sequence before making a recommendation.

State the decision and candidate set.
Specify the quantity to be modeled and list the candidate families or structures before comparing their scores. Record any constraints on parameter values or model fitting.

Fix the data basis.
Identify the observations, exclusions, weights or exposure treatment, and any truncation or censoring. Use the same underlying data and likelihood basis for every candidate in the comparison.

Record fitted quantities.
For each model, record the maximized log-likelihood, the number $k$ of freely estimated parameters, and, for BIC, the chosen $n$ and its meaning. Preserve enough precision to reproduce the scores.

Calculate both criteria consistently.
Apply the same formulas to each candidate. Keep the sign of the log-likelihood visible: a negative log-likelihood contributes a positive value to $-2\ell(\hat{\theta})$.

Check fit and actuarial relevance.
Review graphical and numerical diagnostics, including tail behavior where relevant, and consider the consequences of the model choice for the intended application. A smaller criterion does not establish adequate fit.

Document the decision.
State which criterion or other evidence influenced the choice, identify any disagreement, and explain the practical rationale. Record limitations and the exact inputs so the comparison can be rerun.

Why the Penalties Matter

The criterion formulas combine fit and complexity. For two candidate models $A$ and $B$, the AIC difference is $$ \mathrm{AIC}_B-\mathrm{AIC}_A =-2\{\ell_B(\hat{\theta}_B)-\ell_A(\hat{\theta}_A)\} +2(k_B-k_A). $$ The first term rewards an improvement in maximized log-likelihood; the second charges for additional parameters. Thus, adding a parameter is not automatically beneficial: the likelihood gain must be large enough to offset the penalty. The same algebra for BIC gives a complexity penalty of $(k_B-k_A)\ln(n)$, which depends on the sample size.

One justification for BIC’s penalty comes from approximating the marginal likelihood for a regular model. Under conditions such as a fixed-dimensional parameter vector, an interior maximum, and a smooth likelihood with nonsingular information, a Laplace approximation gives $$ \log \int L(\theta)\pi(\theta)\,d\theta =\ell(\hat{\theta})-\frac{k}{2}\ln(n)+O(1). $$ Multiplying by $-2$ yields $-2\ell(\hat{\theta})+k\ln(n)+O(1)$. BIC retains the terms that grow with sample size and omits the bounded remainder. This approximation is not a guarantee that BIC selects the best model for every predictive or actuarial purpose; its assumptions and intended interpretation matter.

Worked Example: Comparing Two Severity Models on the Same Sample

A hypothetical insurer fits two continuous severity models to the same 80 fully observed losses, using maximum likelihood. Model A has maximized log-likelihood $-120.4$ and $k=2$; Model B has maximized log-likelihood $-116.9$ and $k=3$. Calculate AIC and BIC, then state what the scores do and do not establish.

Method. Use the same $n=80$ and the stated parameter counts for both models. First, $-2\ell_A=240.8$ and $-2\ell_B=233.8$.

For AIC, Model A has $240.8+2(2)=244.8$. Model B has $233.8+2(3)=239.8$. Thus AIC favors Model B by $244.8-239.8=5.0$.

For BIC, $\ln(80)=4.3820266$. Model A has $240.8+2(4.3820266)=249.5641$. Model B has $233.8+3(4.3820266)=246.9461$. Rounded to three decimals, the BIC values are 249.564 and 246.946, so BIC also favors Model B.

Check. Model B improves the maximized log-likelihood by $3.5$, so its fit term improves by $2(3.5)=7.0$. Its extra parameter adds 2.0 to AIC and about 4.382 to BIC. Both improvements exceed their respective penalties. The scores rank Model B lower, but diagnostics—especially relevant tail checks—and the intended actuarial use must still be reviewed. This calculation avoids the mistake of treating the lower score as proof of adequate fit.

Recording Disagreement Rather Than Hiding It

AIC and BIC can prefer different candidates because their complexity penalties differ. Such disagreement is information to explain, not a calculation error by itself. AIC’s penalty is $2k$, whereas BIC’s penalty is $k\ln(n)$; when $\ln(n)>2$, BIC penalizes each additional parameter more heavily. The resulting choice should be considered alongside the purpose of the model, fit diagnostics, and consequences for the actuarial quantity being estimated.

Worked Example: AIC and BIC Give Different Rankings

A hypothetical portfolio has 60 independent annual claim-count observations. A one-parameter Poisson model has maximized log-likelihood $-80.0$. A two-parameter alternative has maximized log-likelihood $-78.8$. The models use the same observations and a comparable likelihood. Calculate both criteria and give a documented interpretation.

For the Poisson model, $-2\ell=160.0$, $k=1$, and $\ln(60)=4.0943446$. Its AIC is $160.0+2=162.0$, and its BIC is $160.0+4.0943446=164.0943$.

For the alternative, $-2\ell=157.6$ and $k=2$. Its AIC is $157.6+4=161.6$. Its BIC is $157.6+2(4.0943446)=165.7887$. To three decimals, the alternative has AIC 161.600 and BIC 165.789.

AIC favors the two-parameter alternative by $162.0-161.6=0.4$, while BIC favors the Poisson model by $165.7887-164.0943=1.6944$. The alternative improves the fit term by $160.0-157.6=2.4$, enough to offset AIC’s extra penalty of 2.0, but not BIC’s extra penalty of 4.0943. The workflow does not pick a winner mechanically: it records the purpose, checks whether the additional structure is supported by diagnostics, and explains which trade-off is appropriate. This avoids reporting only the criterion that supports a preferred model.

What to Record in the Selection File

A compact comparison table is useful, but it should be accompanied by the modeling choices that produced its entries. At a minimum, preserve:

The decision being supported and the full candidate set considered.
The exact data basis, including the observation unit and how incomplete observations were handled.
For every candidate, the maximized log-likelihood, freely estimated parameter count, AIC, and BIC.
The BIC value of $n$, what it counts, and why that convention is appropriate for the likelihood used.
The diagnostic findings and any limitations material to the intended use.
The final rationale, including criterion disagreement and consequences of choosing among the plausible models.

Keep enough digits in intermediate calculations to prevent rounding from changing close rankings. Report rounded scores for communication, but calculate differences from unrounded values. If the observations, likelihood definition, or parameterization changes, treat the resulting comparison as a new analysis and retain the earlier record rather than silently replacing it.

Worked Example: A Selection Record for Incomplete Loss Observations

A hypothetical benefit program has 40 reported loss records, some of which are right-censored at a policy reporting threshold. Two candidate severity models are fitted by likelihood methods that both account for the observed censoring. Model C has maximized log-likelihood $-51.2$ and $k=2$; Model D has maximized log-likelihood $-49.7$ and $k=3$. For this comparison, take $n=40$ observed records. Calculate the scores and state the reproducibility checks.

For Model C, $-2\ell=102.4$. Its AIC is $102.4+2(2)=106.4$. Since $\ln(40)=3.6888795$, its BIC is $102.4+2(3.6888795)=109.7778$.

For Model D, $-2\ell=99.4$. Its AIC is $99.4+2(3)=105.4$, and its BIC is $99.4+3(3.6888795)=110.4666$. Rounded to three decimals, the BIC values are 109.778 and 110.467. AIC favors Model D by 1.0, while BIC favors Model C by about 0.689.

A reproducible record states that $n=40$ means observed records, confirms both likelihoods use the same records and censoring treatment, and verifies $k$ counts every freely estimated parameter in each model. It also records diagnostics relevant to the reporting threshold and the intended benefit calculation. The scores alone do not settle whether either model describes the censored tail adequately. This example avoids comparing a censored-data likelihood with a likelihood that treats censored amounts as fully observed.

Common Workflow Traps

Mixing data bases. Comparing likelihoods from different observations or different censoring treatments makes the score difference difficult or impossible to interpret.
Counting parameters inconsistently. A parameter estimated in one model cannot be treated as fixed in its penalty merely because it is not the parameter of primary interest.
Using the wrong BIC sample size. State what $n$ counts. Do not substitute a total exposure or number of individual claims for the observational unit without justifying that convention for the likelihood.
Choosing by score alone. Information criteria compare relative trade-offs within the candidate set; they do not certify absolute fit, tail adequacy, or suitability for every actuarial decision.
Rounding before ranking. Close scores can appear tied after rounding. Compare unrounded values and then report sensible precision.

Key takeaway: A defensible information-criterion decision begins with a fixed candidate set and a common data basis. It records the likelihood, parameter count, and BIC sample-size convention, checks the numerical ranking against diagnostics, and explains the actuarial rationale and limitations.

Check Your Understanding

For each problem, use the stated data basis and show enough intermediate work to make the comparison reproducible.

Two models use the same 50 observations. Model A has $\ell=-40$ and $k=1$; Model B has $\ell=-38.5$ and $k=2$. Calculate both AIC values and state which is lower. (Answer: AIC A $=82.000$, AIC B $=81.000$; Model B is lower.)
A model has $\ell=-82$, $k=2$, and $n=60$. Calculate AIC and BIC, rounding to three decimals. (Answer: AIC $=168.000$; BIC $=172.189$, or $172.2$ to one decimal.)
Two candidates share the same 40 observations. Model R has $\ell=-50$, $k=1$; Model S has $\ell=-48.8$, $k=2$. Which does AIC prefer? (Answer: AIC R $=102.000$, AIC S $=101.600$; AIC prefers Model S.)
A candidate has a lower AIC than its competitor, but its fitted upper tail is inconsistent with the diagnostic evidence relevant to the intended high-limit application. What should the selection record conclude? (Answer: Record the lower AIC as a relative ranking, document the tail concern, and do not treat the score alone as sufficient justification for selecting the candidate.)
Two models have been fitted to different sets of loss records, and their reported BIC values are compared directly. Identify the central workflow defect and the corrective action. (Answer: The comparison does not share a common data basis; refit or otherwise compare candidates on the same eligible observations and document a consistent likelihood and $n$ convention.)

What You'll Learn