What the Vuong Statistic Says
A Vuong test compares two models by looking at how their fitted probability densities score the same observations. Its result has two parts that must be read together: the sign indicates the direction of the comparison, while the size indicates whether the observed direction is statistically decisive under the test’s assumptions.
Throughout this tutorial, call the first candidate model \(M_1\) and the second \(M_2\). That order matters: reversing the labels reverses the sign of the statistic. The comparison here is for distinguishable, non-nested models, as in the earlier material on non-nested severity models. Both models must be evaluated using the same observations and comparable likelihoods.
The usual two-sided test asks whether the models have equal expected pointwise log-likelihood under the comparison, \(H_0:E[m_i]=0\), against \(H_1:E[m_i]\ne 0\). Under the null and the relevant regularity conditions, \(V\) is approximately standard normal for a sufficiently large sample. The test is intended for distinguishable, non-nested models; the interpretation below is not a general rule for nested likelihood-ratio tests.
Read Direction and Significance Separately
At a 5% two-sided significance level, the standard normal critical values are approximately \(-1.96\) and \(1.96\). Thus, a statistic above \(1.96\) is evidence favoring \(M_1\), and a statistic below \(-1.96\) is evidence favoring \(M_2\). A statistic between those cutoffs does not provide decisive evidence for either model.
| Vuong result at the 5% two-sided level | Interpretation |
|---|---|
| \(V>1.96\) | Statistically significant evidence favors \(M_1\), the first model. |
| \(V<-1.96\) | Statistically significant evidence favors \(M_2\), the second model. |
| \(-1.96\le V\le 1.96\) | The test is inconclusive; neither model is decisively favored. |
This decision rule separates direction from strength of evidence. For example, \(V=0.8\) points toward \(M_1\), but it is not statistically significant at this level. It is not appropriate to report that \(M_1\) “wins” just because the sign is positive. Likewise, \(V=-2.4\) favors \(M_2\), not \(M_1\).
A p-value gives another way to report the evidence. For the two-sided test, it is the probability, under the null and the test’s approximation, of seeing a statistic at least as far from zero as the observed statistic. A small p-value supports a difference in expected log-likelihood; its sign still has to be checked to identify the favored model. A p-value is not the probability that either model is true.
Why the Sign Gives the Direction
The direction follows directly from the definition of \(m_i\). If \(m_i>0\), then the fitted likelihood contribution for observation \(i\) is greater under \(M_1\); if \(m_i<0\), it is greater under \(M_2\). Averaging these differences summarizes the direction across observations.
For each observation, subtract the log density under \(M_2\) from the log density under \(M_1\), using that same observation in both models.
A positive \(\bar m\) means the observed average contribution is larger for \(M_1\); a negative \(\bar m\) means it is larger for \(M_2\).
The factor \(\sqrt n/s_m\) scales the average by its estimated sampling variability. It changes the magnitude, not the direction, because both factors are positive.
Compare \(V\) with the chosen critical values, or use its p-value, before stating whether the directional result is statistically decisive.
This explains why the model order must be written down before interpreting the result. If the numerator were instead defined as \(\log f_2-\log f_1\), every \(m_i\) and \(V\) would change sign, while the underlying comparison would be unchanged.
Worked Example: A Significant Positive Statistic
A hypothetical insurer compares two non-nested severity models for the same \(100\) observed claim amounts. For \(60\) claims the pointwise log-likelihood difference \(m_i=\log f_1-\log f_2\) is \(0.20\); for the remaining \(40\) claims it is \(-0.10\). Calculate the Vuong statistic and interpret it using a 5% two-sided test. This method directly checks both the sign convention and the significance threshold.
Step 1: Calculate the average difference. $$ \bar m=\frac{60(0.20)+40(-0.10)}{100} =\frac{12-4}{100}=0.08. $$
Step 2: Calculate the sample standard deviation. The deviations from the mean are \(0.20-0.08=0.12\) for the first group and \(-0.10-0.08=-0.18\) for the second. Therefore, $$ s_m^2=\frac{60(0.12)^2+40(-0.18)^2}{99} =\frac{0.864+1.296}{99} =\frac{2.16}{99}=0.021818. $$ Thus \(s_m=\sqrt{0.021818}=0.14771\).
Step 3: Standardize. $$ V=\frac{\sqrt{100}(0.08)}{0.14771} =\frac{0.80}{0.14771}=5.416. $$
Since \(5.416>1.96\), the result is statistically significant at the 5% two-sided level. The sign is positive, so the test favors \(M_1\), the first model. The two-sided p-value is less than \(0.0001\). The conclusion is evidence favoring \(M_1\) in this comparison, not proof that \(M_1\) is the true severity distribution.
Worked Example: A Significant Negative Statistic
A hypothetical analyst compares a lognormal model \(M_1\) with a Weibull model \(M_2\) for the same \(64\) recorded losses. The pointwise differences are \(-0.15\) for \(48\) losses and \(0.10\) for \(16\) losses. Calculate and interpret \(V\) at the 5% two-sided level. The method avoids treating a negative statistic as evidence for the first-listed model.
Step 1: Find the mean. $$ \bar m=\frac{48(-0.15)+16(0.10)}{64} =\frac{-7.2+1.6}{64} =-0.0875. $$
Step 2: Find the sample standard deviation. The deviations from \(-0.0875\) are \(-0.0625\) and \(0.1875\): $$ s_m^2=\frac{48(-0.0625)^2+16(0.1875)^2}{63} =\frac{0.1875+0.5625}{63} =\frac{0.75}{63}=0.011905. $$ Therefore, \(s_m=\sqrt{0.011905}=0.10911\).
Step 3: Calculate the statistic. $$ V=\frac{\sqrt{64}(-0.0875)}{0.10911} =\frac{-0.7000}{0.10911}=-6.416. $$
Because \(-6.416<-1.96\), the result is significant and favors \(M_2\), the Weibull model. The two-sided p-value is less than \(0.0001\). The common error would be to see that the statistic is large in absolute value and declare \(M_1\) favored; the negative sign instead points to the second model.
Worked Example: A Positive but Inconclusive Statistic
For \(100\) hypothetical policyholder losses, two fitted models produce \(m_i=0.27\) for \(50\) observations and \(m_i=-0.23\) for the other \(50\). Calculate \(V\) and state what can be concluded at the 5% two-sided level. This calculation distinguishes a favorable direction from decisive evidence.
Step 1: Calculate the mean. $$ \bar m=\frac{50(0.27)+50(-0.23)}{100} =\frac{13.5-11.5}{100}=0.02. $$
Step 2: Calculate the standard deviation. Each value is \(0.25\) away from the mean, so $$ s_m^2=\frac{100(0.25)^2}{99} =\frac{6.25}{99}=0.063131, \qquad s_m=\sqrt{0.063131}=0.25126. $$
Step 3: Calculate \(V\). $$ V=\frac{\sqrt{100}(0.02)}{0.25126} =\frac{0.20}{0.25126}=0.796. $$
The statistic is positive, so the observed average points toward \(M_1\). However, \(0.796\) lies between \(-1.96\) and \(1.96\); the two-sided p-value is approximately \(0.426\). The result is inconclusive: it does not decisively favor either model, and it does not establish that the models are equally good.
Common Interpretation Traps
A sound conclusion names the model order, the direction, and the significance decision. It should not turn a comparison into a broader claim than the test supports.
- Reversing the sign. With \(m_i=\log f_1-\log f_2\), positive values point toward \(M_1\), and negative values point toward \(M_2\). Check the subtraction order before interpreting \(V\).
- Using only the sign. A positive statistic below the critical value is not decisive evidence for \(M_1\). Report direction and statistical significance separately.
- Calling an inconclusive result a tie. Failure to reject equal expected log-likelihood is not proof of equality. The data may simply be too variable or too limited to distinguish the models.
- Misreading a small p-value. It is evidence against the null of equal expected log-likelihood under the test’s assumptions; it is not the probability that the favored model is false, nor a measure of practical importance.
- Comparing unlike likelihoods. The pointwise contributions must be computed for the same observations under both models. A direction based on different data sets or incompatible likelihood contributions is not the stated comparison.
Check Your Understanding
For each problem, use a 5% two-sided test unless stated otherwise. Interpret the sign and significance as separate parts of the conclusion.
- For \(n=100\), the fitted pointwise differences have \(\bar m=0.06\) and \(s_m=0.30\). Calculate \(V\) and interpret it. (Answer: \(V=2.000\); significant positive result, favoring \(M_1\) at 5%.)
- For \(n=64\), \(\bar m=-0.04\) and \(s_m=0.16\). Calculate \(V\), and state whether the result is decisive. (Answer: \(V=-2.000\); significant negative result, favoring \(M_2\) at 5%.)
- For \(n=144\), \(\bar m=0.015\) and \(s_m=0.18\). Find \(V\) and give the correct conclusion. (Answer: \(V=1.000\); the direction points toward \(M_1\), but the result is inconclusive.)
- A comparison reports \(V=-2.35\) using \(m_i=\log f_1-\log f_2\). Which model is favored at the 5% two-sided level, and why? (Answer: \(M_2\), the second model; the statistic is below \(-1.96\).)
- A test gives \(V=1.40\) and a two-sided p-value above \(0.05\). A report says, “The first model is proven better because \(V\) is positive.” Identify the error and give a defensible conclusion. (Answer: Positive indicates direction only; the result is not statistically significant, so neither model is decisively favored.)