Introduction
A likelihood comparison is not a conclusion by itself. A statistic or p-value becomes useful only when the reader can tell which models were compared, what evidence the calculation measures, how that evidence was calibrated, and what assumptions make the calibration reasonable. An actuarial conclusion should also explain whether the statistical result matters for the intended use, such as estimating expected claim cost or assessing tail exposure.
This tutorial develops a disciplined way to report likelihood-based conclusions. It covers nested-model likelihood-ratio tests, comparisons of distinguishable non-nested models using the Vuong statistic, and bootstrap calibration when a standard reference distribution is unsuitable. These methods answer different questions; their statistics and p-values should not be presented as interchangeable measures of model truth.
What a Defensible Conclusion Needs
Before interpreting a result, state the model comparison precisely. For a nested likelihood-ratio test, the null model is the restricted model and the alternative is the encompassing model. The alternative includes all distributions allowed by the null and at least one additional free parameter. For a non-nested comparison, each candidate model has its own parameterization; neither model is a special case of the other.
A complete report should distinguish the following:
- Hypotheses: name the null and alternative models, including relevant restrictions.
- Evidence: report the statistic, its direction if applicable, and the observed likelihood information used to obtain it.
- Calibration: say what reference distribution or simulation procedure produced the p-value.
- Assumptions: identify conditions such as independent observations, a common data set and observation scheme, and regularity conditions for an asymptotic test.
- Meaning: state what the result implies for this comparison, and what it cannot establish about other models or future performance.
A p-value is a probability calculated under a specified null model and its assumptions: it is the probability, if that null model and those assumptions hold, of obtaining a test statistic at least as extreme as the observed statistic. It is not the probability that the null model is true, nor the probability that the result occurred “by chance.”
Why the Calibration Matters
For nested models fitted by maximum likelihood to the same observations, write \(\ell_0\) and \(\ell_1\) for the maximized log likelihoods under the null and alternative, respectively. The likelihood-ratio statistic is
Because the alternative contains the null, maximizing over its larger parameter space cannot produce a lower maximized likelihood, so \(G^2\geq 0\). Under the null and standard regularity conditions, \(G^2\) is approximately chi-square with degrees of freedom equal to the difference in the number of free parameters. This asymptotic result is often called Wilks’ theorem.
The basic reason for the form of the statistic is that the likelihood ratio compares the best fit allowed by the null with the best fit allowed by the alternative. Taking its logarithm converts the ratio into a difference of log likelihoods; multiplying by two gives the conventional statistic whose null distribution has the chi-square approximation under regularity. This calibration is not automatic: boundary parameters, unidentified parameters, very small samples, dependence, or an incorrect likelihood can invalidate or weaken it.
Worked Example: Reporting a Nested Severity-Model Test
A hypothetical insurer fits two severity models to the same 120 independent, complete claim amounts. The null model is exponential, with one free parameter. The alternative is Weibull, with two free parameters; its shape-one restriction gives the exponential model. The maximized log likelihoods are \(\ell_0=-123.0\) and \(\ell_1=-120.8\). Assume the null parameter is interior and the usual regularity conditions hold. At a 5% significance level, what conclusion is defensible?
Method. Use the nested likelihood-ratio test. The models use the same observations and are nested, and the specified regularity assumptions support the chi-square approximation.
Calculate the statistic. The log-likelihood difference is \( -120.8-(-123.0)=2.2\). Therefore, \(G^2=2(2.2)=4.4\). The difference in free parameters is \(2-1=1\), so the reference distribution is approximately chi-square with one degree of freedom.
Calibrate the evidence. The upper-tail probability is \(P(\chi^2_1\geq 4.4)\approx 0.0359\). Since \(0.0359<0.05\), reject the exponential restriction at the 5% level under the stated assumptions.
Conclusion and meaning. The data provide evidence against the exponential restriction relative to this Weibull alternative. This result does not prove that the Weibull model is true, or that it is adequate among all possible severity models. Before changing a pricing or reserving assumption, an actuary should also examine the fitted implications for quantities relevant to the decision, such as expected payment or tail losses.
Common mistake avoided. Reporting only that the Weibull likelihood is larger omits the size and calibration of the improvement. Here the test uses twice the log-likelihood difference, not the difference alone, and its degrees of freedom are the one additional free parameter.
Conclusions for Non-Nested Comparisons
For distinguishable, non-nested models fitted to the same independent observations, the Vuong comparison uses per-observation log-likelihood differences. Let \(d_i\) be the log likelihood contribution under model A minus that under model B for observation \(i\). A positive average difference favors A in fitted log likelihood; a negative one favors B. The standardized Vuong statistic divides the average difference by its estimated standard error. Under the method’s assumptions, it is compared with a standard normal distribution.
The direction and statistical evidence must be reported together. A positive statistic points toward A, but a small absolute statistic does not provide strong evidence that A is better. This comparison requires the models to be distinguishable in the relevant sense; when they overlap or are effectively indistinguishable, the usual standard-normal calibration may not apply. The analyst must also use the same observations and account consistently for censoring, truncation, or other features of the observation process.
Worked Example: A Positive Likelihood Difference Without Strong Evidence
For 50 independent, hypothetical claim amounts observed under the same sampling scheme, two distinguishable non-nested severity models are fitted. For each claim, define \(d_i\) as model A’s log likelihood contribution minus model B’s. The sample mean of the \(d_i\) values is \(0.096\), and their sample standard deviation is \(1.2\). Use the two-sided standard-normal calibration at the 5% level.
Method. Use the Vuong statistic, assuming the observations are independent, both models are fitted to the same data, and the models are distinguishable so that the stated standard-normal calibration is appropriate.
Standardize the average difference. Its estimated standard error is \(1.2/\sqrt{50}=1.2/7.0710678\ldots=0.1697056\ldots\). Thus \(V=0.096/0.1697056\ldots=0.565685\ldots\), which rounds to \(0.566\).
Calculate the two-sided p-value. Using the unrounded statistic, \(p=2[1-\Phi(0.565685\ldots)]\approx0.5716\). The statistic is positive, so the observed average log likelihood difference favors A, but the p-value is much larger than \(0.05\). There is not strong evidence from this test to distinguish the models at the 5% level.
Interpret the likelihood ratio carefully. The total log-likelihood difference is \(50(0.096)=4.8\), so the ratio of the two maximized likelihoods is \(\exp(4.8)\approx121.51\) in favor of A. This raw ratio does not replace the Vuong calibration: its size does not account for the variability of the per-observation differences.
Conclusion and common mistake avoided. State that A has the larger fitted likelihood on these data, but the Vuong test does not provide strong evidence that it is superior. Do not call the p-value the probability that A is no better than B. Also avoid reporting a p-value computed from a rounded statistic as if it used the unrounded value: at \(V=0.566\), the two-sided value is about \(0.5714\); using the unrounded statistic gives about \(0.5716\).
Bootstrap Calibration and Practical Interpretation
A parametric bootstrap can estimate a test statistic’s null distribution when a standard asymptotic reference distribution is not reliable. The procedure simulates data under the fitted null model, applies the same observation mechanism as in the actual data, refits the models to every simulated data set, and recalculates the statistic. The resulting simulated statistics approximate the distribution expected under the null, conditional on the model and assumptions used to generate the simulations.
For a likelihood-ratio statistic, if \(B\) of \(M\) simulated statistics are at least as large as the observed statistic, a commonly used finite-simulation estimate of the upper-tail probability is \((B+1)/(M+1)\). The added one prevents a reported probability of zero when no simulated statistic exceeds the observed value. The estimate also has simulation error: when the result is near a decision threshold, more replicates may be needed to judge how stable the estimate is.
Worked Example: Qualifying a Bootstrap-Calibrated Conclusion
A hypothetical portfolio contains independently observed claim amounts, some of which are right-censored at policy-specific thresholds. An actuary compares a restricted model with a nested model whose additional parameter is on a boundary under the null, so the usual chi-square calibration is not assumed to be reliable. The observed likelihood-ratio statistic is \(3.6\). In \(M=2000\) parametric-bootstrap samples generated under the fitted null, with the censoring mechanism replicated and both models refitted each time, \(B=86\) statistics are at least \(3.6\). Use a 5% threshold.
Method. Use the null parametric bootstrap because the boundary condition makes the standard chi-square calibration questionable. The conclusion depends on the null model, the simulation mechanism, the fitting procedure, and the accuracy of the bootstrap implementation.
Estimate the tail probability. Applying the finite-simulation correction gives \(p=(86+1)/(2000+1)=87/2001=0.04348\ldots\), or \(0.0435\) to four decimal places. This is below \(0.05\), so the observed statistic is unusual under the simulated null distribution at the stated threshold.
Assess simulation precision. The approximate Monte Carlo standard error is \(\sqrt{0.04348(1-0.04348)/2001}\approx0.0046\). Since this uncertainty is not negligible compared with the difference between \(0.0435\) and \(0.05\), the result should be described as close to the threshold rather than as overwhelming evidence. Additional bootstrap replicates could clarify the stability of the estimated tail probability.
Conclusion and meaning. Under the specified null model and bootstrap procedure, the data provide evidence against the restricted model at the 5% level, but the evidence is borderline and subject to simulation uncertainty. The comparison does not prove that the larger model is true. An actuarial recommendation should additionally assess whether the model change materially affects estimates or decisions relevant to the portfolio.
Common mistake avoided. Do not use a chi-square p-value merely because the statistic is called a likelihood ratio. The calibration must reflect the stated boundary condition and observation process; here, the bootstrap reproduces censoring and refits both models.
Common Traps in Writing Conclusions
A technically correct calculation can still lead to an indefensible statement. Separate what the data show from what the model comparison cannot establish.
- “The larger likelihood proves the model is true.” Maximized likelihood is relative to the candidate models and observed data. A larger value is not proof of truth or adequacy.
- “The p-value is the probability the null is true.” It is a tail probability under the null and the test assumptions, not a probability assigned to a model.
- Omitting the hypotheses. “The result is significant” is incomplete unless the null restriction and alternative are clear.
- Leaving out calibration and assumptions. Say whether the p-value came from a chi-square approximation, a standard-normal Vuong test, or a parametric bootstrap, and identify the conditions that justify it.
- Confusing statistical and actuarial importance. A statistically detectable difference may have little effect on an estimate or decision; a practically important tail difference may also warrant careful investigation even when a test is inconclusive.
- Ignoring the data used. Likelihoods must be based on comparable observations and a correctly specified observation mechanism. Censoring or truncation handled differently across models can make the comparison misleading.
Check Your Understanding
For each problem, state the relevant evidence and the limit of the conclusion. Use the stated calibration assumptions.
- A one-parameter null model is nested in a two-parameter alternative. Their maximized log likelihoods are \(-84.0\) and \(-82.5\), respectively. Under regularity conditions, calculate the likelihood-ratio statistic and identify its chi-square degrees of freedom. (Answer: \(G^2=3.0\), with 1 degree of freedom; the result is not proof that the alternative is true.)
- For 64 independent observations, a non-nested comparison has mean per-observation log-likelihood difference, A minus B, of \(-0.10\), with sample standard deviation \(0.8\). Calculate the Vuong statistic. (Answer: standard error \(=0.8/\sqrt{64}=0.10\), so \(V=-1.00\); the direction favors B, but the statistic alone does not establish strong evidence.)
- A parametric bootstrap produces \(M=999\) null statistics, of which \(B=49\) are at least as large as the observed statistic. Calculate the corrected tail estimate. (Answer: \((49+1)/(999+1)=0.0500\); interpret it under the bootstrap assumptions, not as the probability that the null is true.)
- A nested test gives \(G^2=5.0\) with 2 degrees of freedom, and the chi-square calibration is justified. The 5% critical value is 5.991. What is the decision at the 5% level? (Answer: do not reject the null, because \(5.0<5.991\); this does not establish that the null model is true.)
- Two models are fitted to different subsets because one analyst drops censored claims and the other includes them. Can their maximized likelihoods support the usual direct likelihood comparison? (Answer: not without addressing the difference in data and observation scheme; a defensible comparison requires likelihoods defined for comparable data under appropriately handled censoring.)