Why Non-Nested Comparisons Need Care
A severity sample may be reasonably described by more than one distributional family. For example, an exponential model and a Pareto model can both assign positive density to large losses, but neither family is generally a special case of the other. A larger maximized likelihood by itself favors a more flexible fit, while a standard likelihood-ratio test does not provide a valid comparison when the models are non-nested.
Two useful alternatives are information criteria, which balance fit against the number of estimated parameters, and out-of-sample log scores, which assess predictions on observations not used to fit the models. Both methods require a comparable likelihood or density basis: the models should be fitted to the same observations under the same sampling and observation scheme.
AIC and BIC are relative selection tools, not goodness-of-fit tests. AIC is motivated by estimating relative out-of-sample information loss, with a penalty for estimating parameters. BIC uses a different penalty that grows with sample size and, under regularity conditions, is related to an approximation to model evidence. Neither criterion establishes that the winning model is true, and their rankings need not agree.
Likelihoods and the Non-Nested Test Trap
A likelihood-ratio statistic is often written \(2(\ell_1-\ell_0)\), where model 0 is a restricted model nested within model 1. Under suitable regularity conditions, the statistic can then be compared with a chi-squared reference distribution whose degrees of freedom reflect the number of restrictions. The nesting condition matters: the null model’s possible distributions must be included among those of the alternative.
For example, an exponential severity distribution is not generally obtained by selecting a parameter value within the Pareto family, nor is the Pareto family generally obtained by restricting an exponential model. Their maximized likelihoods can be compared through AIC, BIC, or predictive scoring, but the usual nested-model chi-squared likelihood-ratio calibration does not follow. A numerical difference in log likelihood does not repair this problem.
Worked Example: AIC and BIC for Exponential and Pareto Fits
Worked Example: AIC and BIC for Exponential and Pareto Fits
A hypothetical analyst observes four exact, independent severities, in thousands of dollars: \(1,2,3,4\). Compare an exponential model with rate \(\lambda\) to a Type I Pareto model with fixed lower endpoint \(x_m=1\) and unknown shape \(\alpha\). Use maximum likelihood and calculate AIC and BIC. Assume the observations are uncensored and that the support conditions hold.
Method and assumptions. For the exponential model, \(f(x)=\lambda e^{-\lambda x}\), \(x\geq 0\), with one estimated parameter. For the Pareto model, \(f(x)=\alpha x_m^\alpha x^{-(\alpha+1)}\), \(x\geq x_m\); here \(x_m=1\) is fixed, so there is also one estimated parameter. The sample satisfies \(x_i\geq1\). The log likelihoods are maximized separately on the same four observations.
Exponential fit. The sample total is \(1+2+3+4=10\), so \(\hat\lambda=n/\sum x_i=4/10=0.4\). Thus
With \(k=1\) and \(n=4\),
and
Pareto fit. Since \(x_m=1\), the log likelihood is \(\ell(\alpha)=4\ln\alpha-(\alpha+1)\sum_i\ln x_i\). The product of the four observations is \(24\), so \(\sum_i\ln x_i=\ln24\). Differentiating gives \(4/\alpha-\ln24=0\), hence \(\hat\alpha=4/\ln24\approx1.2586319\). The second derivative is \(-4/\alpha^2<0\), confirming this stationary point is a maximum. Substitution gives
Using the unrounded estimate in the calculation, the criteria are
and
Conclusion and check. Both criteria are lower for the Pareto fit on this sample. The comparison uses \(k=1\) for each model and \(n=4\) for each BIC. It does not establish that the Pareto distribution is correct; it says only that these criteria prefer it among these two specified fits. The method avoids treating the two non-nested families as if a standard likelihood-ratio test applied.
Held-Out Log Scores
A log score evaluates the density a fitted model assigns to an observation. If a model fitted on training data has density \(f\), and the independent test observations are \(x_1,\ldots,x_m\), the total and average test log scores are
For independent observations, their joint density is the product of their individual densities. Taking the logarithm turns that product into a sum, which is why individual log-density contributions add. Larger log scores are preferred: they correspond to greater density for the observed test sample. A log density is not a probability and need not be negative.
The training sample must not also serve as the test sample for an honest out-of-sample assessment. A single split can be variable, especially with a small sample; repeated splits or cross-validation can give a more stable comparison. The scoring rule must also match the observed data. For censored or truncated severities, one must use the appropriate likelihood contributions, not pretend every recorded value is an exact uncensored loss.
Worked Example: Comparing Held-Out Scores
Use the exponential and fixed-endpoint Pareto fits from the preceding example, fitted on the training observations \(1,2,3,4\) in thousands of dollars. The separate test observations are \(2\) and \(5\), also in thousands. Calculate each model’s total and average test log score. The fitted parameters are \(\hat\lambda=0.4\) and \(\hat\alpha=4/\ln24\approx1.258631922\), with \(x_m=1\).
Exponential model. Its log density is \(\ln(0.4)-0.4x\). At \(x=2\), the contribution is \(\ln(0.4)-0.8=-0.9162907-0.8=-1.7162907\). At \(x=5\), it is \(\ln(0.4)-2=-0.9162907-2=-2.9162907\). Therefore
Pareto model. Since \(x_m=1\), its log density is \(\ln\hat\alpha-(\hat\alpha+1)\ln x\). Using the unrounded fitted shape, the contribution at \(x=2\) is approximately \(-1.3355390\), and at \(x=5\) it is approximately \(-3.4051025\). Thus
The exponential score is larger because \(-4.6325815>-4.7406415\); it assigns greater density to this particular held-out pair. This does not contradict the Pareto model’s lower AIC and BIC on the training data. The criteria answer different questions, and two test observations provide limited evidence. The calculation avoids selecting the model merely because its training likelihood was larger.
Units and Interpretation
A density depends on the measurement unit. If a loss is recorded in thousands rather than dollars, the numerical density changes by a scale factor, and its logarithm changes by an additive constant. Consequently, raw log scores from different units must not be compared as though they were on the same scale. If every candidate is evaluated on the same observations in the same units, the common transformation adds the same constant to each model’s score, so the ranking is unchanged.
Worked Example: Checking a Change of Severity Units
For the two test losses \(2\) and \(5\) thousands of dollars, the total scores just calculated are \(-4.6325815\) for the exponential model and \(-4.7406415\) for the Pareto model. Convert the test data to dollars and determine the transformed scores. In dollars, the exponential rate is \(0.0004\) per dollar, and the Pareto endpoint is \(1000\) dollars with the same shape \(\hat\alpha\).
Method. If \(Z=1000X\), the density transformation is \(f_Z(z)=f_X(z/1000)/1000\). Therefore each log-density contribution in dollars is the corresponding contribution in thousands minus \(\ln1000\). For two observations, each model’s total score changes by \(-2\ln1000\), where \(2\ln1000=13.8155106\).
The dollar-scale totals are therefore
and
The exponential score remains larger, since both scores changed by the same amount. The common shift is essential: the numerical value of a density score is unit-dependent, while this same-sample, same-unit ranking is preserved under the common conversion.
Practical Comparison and Common Traps
A disciplined comparison starts by checking that the candidate models describe the same response variable and use the same observations. Confirm support restrictions, parameter counts, and whether the data are exact, censored, truncated, or grouped. Then calculate AIC and BIC from maximized likelihoods, and use held-out log scores when a suitable independent test set is available. These methods are complementary rather than interchangeable.
- Using a nested-model test for non-nested families. A chi-squared likelihood-ratio reference distribution requires nesting and regularity conditions; it is not justified merely because both models have likelihoods.
- Choosing the larger training likelihood without a penalty. More flexible models can improve their fitted likelihood. AIC and BIC account for parameter count differently.
- Calling the score closer to zero better. This is not a general rule. A score of \(1\) is larger than \(0.5\), even though it is farther from zero. Compare which score is numerically larger.
- Comparing scores across units or different test data. A density score changes under unit conversion. Keep the test observations and units common across models.
- Scoring censored observations as exact losses. A policy-limit observation may contribute a survival probability rather than a density at the limit. Use the likelihood contribution appropriate to how the observation was recorded.
- Reading a winner as proof. AIC, BIC, and held-out scores rank specified candidates; they do not certify a model or replace examination of fit and data assumptions.
Check Your Understanding
For each problem, state the comparison method and keep the likelihood or score basis consistent.
- A hypothetical exponential model is fitted by maximum likelihood to five independent exact losses with total \(20\) (in thousands of dollars). Find its fitted rate and maximized log likelihood. (Answer: \(\hat\lambda=0.25\); \(\ell=5\ln(0.25)-5\approx-11.9315\).)
- Two models fitted to the same \(n=50\) observations have maximized log likelihoods \(-90\) and \(-89\), with \(k=2\) and \(k=4\), respectively. Calculate both AIC values and identify the lower-AIC model. (Answer: \(184\) and \(186\); the first model has lower AIC.)
- An exponential model with fitted rate \(0.5\) assigns density \(f(x)=0.5e^{-0.5x}\) to each of two independent test losses \(1\) and \(3\). Find the total and average test log score. (Answer: total \(\ln(0.5)-0.5+\ln(0.5)-1.5\approx-3.3863\); average \(\approx-1.6931\).)
- A Type I Pareto model has fixed endpoint \(x_m=1\) and fitted shape \(\alpha=2\). Find the log density at a test loss \(x=2\), and state whether that contribution is a probability. (Answer: \(\ln2-3\ln2=-2\ln2\approx-1.3863\); it is a log density, not a probability.)
- Two severity families are non-nested, and one has a larger maximized likelihood. Is the usual chi-squared likelihood-ratio test justified solely by that fact? (Answer: No; the standard test requires a nested-model relationship and suitable regularity conditions.)