The Vuong Test For Distinguishable Models

Why Compare Observation-Level Likelihoods?

Two non-nested severity models can both provide plausible descriptions of the same hypothetical loss data. Comparing their total maximized log-likelihoods says which model assigns the larger overall likelihood to the observed sample, but does not by itself provide a test calibrated to sampling variation. The Vuong test uses the pattern of likelihood differences across observations to assess whether one model has a higher expected log-likelihood than the other.

The test is based on paired contributions: each observation is evaluated under both models, and the difference is calculated for that same observation. This pairing matters. The models must be fitted to the same observations, with likelihood contributions defined on the same basis. The resulting statistic is approximately standard normal under its null hypothesis when the models are distinguishable and the required regularity conditions hold.

Definition: Let $X_1,\ldots,X_n$ be independent observations from a common distribution $g$. Consider two non-nested models with densities or probability mass functions $f_1(x;\theta_1)$ and $f_2(x;\theta_2)$, fitted to the same data. Let $\hat{\theta}_1$ and $\hat{\theta}_2$ be their respective maximum-likelihood estimates. The observation-level log-likelihood difference is $$ d_i=\log f_1(X_i;\hat{\theta}_1)-\log f_2(X_i;\hat{\theta}_2). $$ A positive $d_i$ means model 1 assigns a greater likelihood contribution to observation $i$; a negative $d_i$ means model 2 does.

The usual comparison concerns the expected log-likelihood difference under the data-generating distribution. The null hypothesis is $H_0:E_g[d_i]=0$, interpreted asymptotically using the models’ probability limits as their parameter estimates converge. The two-sided alternative is $H_1:E_g[d_i]\ne 0$. A positive expected difference favors model 1 in expected log-likelihood; a negative one favors model 2. This is a comparison of relative fit, not a test that either candidate model is exactly true.

Constructing the Statistic

Calculate the sample mean $\bar d$ and sample standard deviation $s_d$ of the observation-level differences. For a distinguishable pair, the Vuong statistic is

V=\frac{\sqrt{n}\,\bar d}{s_d}, \qquad \bar d=\frac{1}{n}\sum_{i=1}^n d_i, \qquad s_d=\sqrt{\frac{1}{n-1}\sum_{i=1}^n(d_i-\bar d)^2}.

The distinguishability condition is that the population variance of the log-likelihood ratio is positive: $\operatorname{Var}_g(\log\{f_1(X;\theta_1^*)/f_2(X;\theta_2^*)\})>0$, where $\theta_1^*$ and $\theta_2^*$ denote the limiting fitted parameters. In words, the two models must not give the same relative log-likelihood contribution for almost every possible observation. The condition is about variation in the observation-level log ratio, not simply whether the fitted total likelihoods differ.

The standard-normal reference is asymptotic. Its use requires independent, identically distributed observations, comparable likelihood contributions, regular and consistent parameter estimation, a finite variance for the log-likelihood differences, and distinguishability. In applications, the estimated parameters are substituted in the $d_i$'s. If the conditions hold and the null is true, $V$ is approximately $N(0,1)$ for sufficiently large $n$. For a two-sided test at significance level $0.05$, the usual large-sample critical values are approximately $-1.96$ and $1.96$.

Worked Example: Comparing Two Fitted Severity Models

An analyst fits two non-nested severity models to the same six hypothetical losses. The table gives their fitted log-likelihood contributions. Calculate the Vuong statistic and give the 5% two-sided conclusion. Assume the observations and fitted models satisfy the test’s regularity conditions.

Observation	Model 1 contribution	Model 2 contribution	$d_i$
1	-1.6	-2.0	0.4
2	-1.7	-1.5	-0.2
3	-1.9	-2.2	0.3
4	-1.9	-1.8	-0.1
5	-1.9	-2.1	0.2
6	-1.4	-1.4	0.0

Method. Subtract the Model 2 contribution from the Model 1 contribution within each row. The differences are $0.4,-0.2,0.3,-0.1,0.2,0.0$, whose sum is $0.6$. Thus $\bar d=0.6/6=0.1$.

The deviations from $0.1$ are $0.3,-0.3,0.2,-0.2,0.1,-0.1$. Their squared values sum to $0.09+0.09+0.04+0.04+0.01+0.01=0.28$. Therefore,

s_d=\sqrt{\frac{0.28}{6-1}}=\sqrt{0.056}\approx 0.2366.

Using the unrounded standard deviation in the statistic gives $V=\sqrt{6}(0.1)/\sqrt{0.056}\approx 1.035$, rounded to three decimals. Since $|1.035|<1.96$, the result does not reject equal expected log-likelihood at the 5% level. The positive sign points toward Model 1 in this sample, but the evidence is not statistically significant at that level. This does not establish that the models are equally good.

Check. The differences sum to $0.6$, so their mean is positive; the statistic is therefore positive, as calculated. The common mistake avoided here is reversing the subtraction order for some observations.

Why the Normal Reference Is Appropriate

The statistic standardizes the average of the observation-level log-likelihood differences. The basic justification is the central limit theorem: under independence, a finite nonzero variance, and the null hypothesis of zero expected difference, the sample mean of the differences fluctuates around zero on a scale proportional to $1/\sqrt{n}$. Dividing by an estimate of that fluctuation scale produces an approximately standard-normal statistic. For maximum-likelihood fitted models, regularity conditions are also needed for substituting the estimated parameters into the observation-level contributions.

The sign has a direct interpretation because the numerator retains the Model 1 minus Model 2 order. A large positive value is evidence favoring Model 1; a large negative value is evidence favoring Model 2. Under a two-sided test, the decision depends on the magnitude $|V|$, while the sign identifies the direction. A one-sided conclusion would require a directional alternative chosen in advance, rather than selected after seeing the statistic.

Justification: If $g$ is the data-generating density and the relevant expectations exist, then $$ E_g\!\left[\log\frac{f_1(X;\theta_1^*)}{f_2(X;\theta_2^*)}\right] =D_{\mathrm{KL}}(g\Vert f_2)-D_{\mathrm{KL}}(g\Vert f_1), $$ where $D_{\mathrm{KL}}(g\Vert f_j)=E_g[\log\{g(X)/f_j(X;\theta_j^*)\}]$ is the Kullback-Leibler divergence. To see this, subtract the two divergence expressions: the terms $E_g[\log g(X)]$ cancel, leaving $E_g[\log f_1(X;\theta_1^*)-\log f_2(X;\theta_2^*)]$. Thus a positive expected log-likelihood difference corresponds to a smaller divergence for Model 1, or closer expected fit to $g$, among these two candidates. The null equates their expected log-likelihoods; it does not assert that $g$ belongs to either model.

Worked Example: A Two-Sided Test With a Borderline-Looking Statistic

For ten hypothetical severity observations, a fitted comparison gives the following differences, always defined as Model 1 minus Model 2: $0.38,0.38,-0.22,-0.22,0.08,0.08,0.08,0.08,0.08,0.08$. Assume the test conditions hold. Calculate the statistic, approximate two-sided $p$-value, and 5% decision.

The sum is $0.38+0.38-0.22-0.22+6(0.08)=0.80$, so $\bar d=0.80/10=0.08$. Relative to the mean, the deviations are $0.30,0.30,-0.30,-0.30$, and six zeros. Their squared sum is $4(0.30^2)=0.36$. Hence

s_d=\sqrt{\frac{0.36}{9}}=0.20, \qquad V=\frac{\sqrt{10}(0.08)}{0.20}=1.2649\ldots\approx 1.265.

The two-sided standard-normal tail probability is $2[1-\Phi(1.265)]\approx 0.2059$, rounded to four decimal places. Since $0.2059>0.05$, do not reject the null at the 5% level. Model 1 has the positive sample mean difference, but the observed statistic does not provide strong evidence of a difference in expected log-likelihood.

Check. The four nonzero deviations contribute $0.36$, and the remaining six contribute zero, confirming the sample standard deviation. The common mistake avoided is treating a positive statistic as automatically significant; direction and evidence strength are separate.

Worked Example: Zero Observed Variation

A comparison of two fitted models on five hypothetical observations gives differences $0.15,0.15,0.15,0.15,0.15$. Determine whether the usual Vuong statistic can be calculated, and state what conclusion is justified.

The mean difference is $\bar d=5(0.15)/5=0.15$. Each difference equals its mean, so every deviation is zero and the sum of squared deviations is $0$. Thus $s_d=\sqrt{0/(5-1)}=0$. Substitution into $V=\sqrt{n}\bar d/s_d$ would require division by zero, so the usual statistic is undefined.

This sample does not permit a standard-normal Vuong test using the stated formula. It does not, by itself, prove that the model pair is indistinguishable in the population: a small sample can show no observed variation even when the population variance is positive. The population distinguishability assumption must be considered separately. The error to avoid is reporting an infinite statistic and treating it as valid evidence; a zero estimated denominator makes this calibration unavailable.

Distinguishability and Exam Traps

The Vuong test is not the ordinary likelihood-ratio test for nested models. Its normal reference is tied to the observation-level log-likelihood differences and the positive-variance distinguishability condition. Do not compare the statistic with a chi-square reference merely because it was constructed from likelihoods. The earlier material on likelihood-ratio tests addresses the nested-model setting; the non-nested comparison here uses a different standardization and reference distribution.

The models must also use the same observations and the same likelihood basis. For example, a log-likelihood contribution from a density cannot be compared directly with one from a differently defined response or observation mechanism. If a loss is censored or the sample is truncated, the likelihood contributions must reflect that sampling information; using ordinary uncensored density contributions would not meet the comparability requirement.

Set the direction.
Define $d_i$ consistently as Model 1 minus Model 2 for every observation.

Summarize the paired differences.
Calculate $\bar d$ and the sample standard deviation $s_d$, not the standard deviation of the original losses.

Check applicability.
Confirm common observations, a non-nested comparison, suitable regularity conditions, and positive variation in the population log ratio. If the sample $s_d=0$, the statistic is not defined.

Calculate and interpret.
Use $V=\sqrt{n}\bar d/s_d$, compare with the standard-normal reference, and use the sign to identify the direction.

A frequent arithmetic trap is to use the total log-likelihood difference divided by $s_d$ as the statistic. In the ten-observation example, the total difference was $0.80$, so that incorrect calculation gives $0.80/0.20=4.00$. The correct statistic is $0.80/(\sqrt{10}\,0.20)=1.265$, because $\bar d=(\sum_i d_i)/n$ and the statistic multiplies the mean by $\sqrt n$. Another trap is reversing the definition of $d_i$ mid-calculation: that reverses the sign and therefore the stated direction of support, even though the absolute value for a two-sided test is unchanged.

Key takeaway: The Vuong statistic standardizes the mean of paired, observation-level log-likelihood differences. Its approximate standard-normal reference requires a valid common-data comparison and distinguishable models with positive log-ratio variance. Interpret both the magnitude and the direction, and do not use the statistic when its estimated standard deviation is zero.

Check Your Understanding

For each problem, use the stated Model 1 minus Model 2 convention and the approximate standard-normal reference when the assumptions are specified.

For five observations, the differences are $0.5,0.1,0.3,-0.1,0.2$. Calculate $\bar d$, $s_d$, and $V$, then state the two-sided 5% decision. (Answer: $\bar d=0.2000$, $s_d=\sqrt{0.05}\approx0.2236$, $V=2.000$; reject at 5%.)
For four observations, the differences are $0.3,-0.1,0.1,-0.3$. Calculate the statistic and state whether the sample standard deviation is positive. (Answer: $\bar d=0$, $s_d=\sqrt{0.20/3}\approx0.2582$, $V=0$; the sample standard deviation is positive.)
A comparison has $n=9$, $\bar d=-0.12$, and $s_d=0.18$. Calculate $V$, identify the favored direction in the sample, and give the two-sided 5% decision. (Answer: $V=3(-0.12)/0.18=-2.000$; Model 2 is favored by the sign, and reject at 5%.)
A valid distinguishable-model comparison has $n=16$, $\bar d=-0.06$, and $s_d=0.12$. Calculate $V$ and decide at the two-sided 5% level. (Answer: $V=4(-0.06)/0.12=-2.000$; reject at 5%, with the direction favoring Model 2.)
In a sample of six, every observation-level difference is $0.10$. Can the usual statistic be standardized, and does the sample establish population indistinguishability? (Answer: No; $s_d=0$, so $V$ is undefined. No; zero observed variation in this sample alone does not establish zero population variance.)

What You'll Learn