Deriving The Vuong Statistic’s Standard Error

Why the Standard Error Depends on Variation Across Observations

The Vuong comparison summarizes how two candidate models perform relative to each other observation by observation. A positive log-likelihood difference means that the first model assigns a higher likelihood contribution to that observation; a negative difference means the second model does. The average difference describes the comparison on the scale of log likelihood, but that average alone does not say how variable the individual contributions are.

The standard error measures the estimated sampling variation of the average difference. Its calculation depends on the centered variance of the observation-level differences, not their uncentered second moment. This tutorial derives that calculation and works through cases where centering, scaling by the sample size, or using total rather than individual likelihood contributions can change the result.

Definition: Suppose $X_1,\ldots,X_n$ are independent observations and two candidate models have fitted density or probability mass functions $f_1(\,\cdot\,;\hat\theta_1)$ and $f_2(\,\cdot\,;\hat\theta_2)$. Define the observation-level log-likelihood difference $$ m_i=\log f_1(X_i;\hat\theta_1)-\log f_2(X_i;\hat\theta_2),\qquad i=1,\ldots,n. $$ Its sample mean is $\bar m=n^{-1}\sum_{i=1}^n m_i$. The Vuong standard error uses an estimate of the variance of these differences; with the asymptotic $n$-denominator convention, that estimate is $\hat\omega^2=n^{-1}\sum_{i=1}^n(m_i-\bar m)^2$, and the estimated standard error of $\bar m$ is $\widehat{\operatorname{SE}}(\bar m)=\sqrt{\hat\omega^2/n}$.

The log-likelihood contributions must refer to the same observation under both models, evaluated on the same data and with likelihoods defined on compatible supports. For independent observations, a total log likelihood is the sum of its observation-level contributions. The comparison uses the differences of those individual contributions, not just the difference between the two total log likelihoods: the latter gives their sum but not the variation needed for a standard error.

Deriving the Estimated Standard Error

For a fixed pair of candidate models, let $M_i$ denote the random log-likelihood difference from observation $i$. Under an independent, identically distributed sampling model, suppose $E[M_i]=\mu_m$ and $\operatorname{Var}(M_i)=\omega^2<\infty$. The variance of the sample mean follows from independence:

\operatorname{Var}(\bar M) =\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n M_i\right) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(M_i) =\frac{\omega^2}{n}.

The unknown $\omega^2$ is estimated from the observed differences. Each difference is compared with the sample mean, because variance measures spread around the mean. The resulting plug-in estimate of the variance of $\bar m$ is $\hat\omega^2/n$.

Compute the paired differences.
For each observation, subtract the second model’s log-likelihood contribution from the first model’s contribution: $m_i=\log f_1(X_i;\hat\theta_1)-\log f_2(X_i;\hat\theta_2)$.

Find their mean.
Calculate $\bar m=n^{-1}\sum_i m_i$. This is the observed average log-likelihood difference.

Estimate their variance.
Subtract $\bar m$ from each $m_i$, square the deviations, add them, and divide by $n$ for the asymptotic Vuong variance estimate.

Estimate the standard error.
Divide the variance estimate by $n$ and take the square root. Equivalently, divide the square root of the estimated variance of the individual differences by $\sqrt n$.

The centering step is essential. Expanding the squared deviations gives the identity

\frac{1}{n}\sum_{i=1}^n(m_i-\bar m)^2 =\frac{1}{n}\sum_{i=1}^n m_i^2-\bar m^2.

Thus the raw second moment $n^{-1}\sum_i m_i^2$ equals the centered variance plus the square of the mean. It is not the variance unless the sample mean happens to be zero. Treating it as the variance overstates the estimated spread whenever $\bar m\ne 0$, and therefore changes the standard error.

Some statistical texts define the sample variance with denominator $n-1$, making it unbiased for the variance of independent observations under the usual fixed-parameter sampling model. The Vuong asymptotic convention uses $n$ in $\hat\omega^2$. If the alternative $n-1$ convention is used, it must be carried consistently into the standard error; the two estimates become asymptotically equivalent as $n$ grows. Do not combine the $n-1$ variance with a formula that assumes the $n$-denominator version without noting the difference.

Worked Calculations

Worked Example: Centering Five Log-Likelihood Differences

In a hypothetical comparison of two fitted severity models, the five observation-level differences, in data order, are $(-0.2,0,0.1,0.3,0.8)$. Calculate the asymptotic variance estimate, the standard error of the mean, and the standardized statistic $V=\bar m/\widehat{\operatorname{SE}}(\bar m)$. Then calculate the result of the common error of using the raw second moment as the variance.

Method. First find the mean: $$ \bar m=\frac{-0.2+0+0.1+0.3+0.8}{5} =\frac{1.0}{5}=0.2000. $$ The deviations from the mean are $(-0.4,-0.2,-0.1,0.1,0.6)$. Their squared values are $0.16,0.04,0.01,0.01,0.36$, which sum to $0.58$. Therefore, $$ \hat\omega^2=\frac{0.58}{5}=0.1160. $$

The estimated variance of the mean is $0.1160/5=0.0232$, so $$ \widehat{\operatorname{SE}}(\bar m)=\sqrt{0.0232}=0.152315\ldots, \qquad V=\frac{0.2000}{0.152315\ldots}=1.3131. $$ The standardized statistic is $1.3131$ to four decimal places.

Check the trap. The raw second moment is not $0.1240$. Directly, the squared differences sum to $0.04+0+0.01+0.09+0.64=0.78$, so their average is $0.78/5=0.1560$. This agrees with $0.1160+(0.2000)^2=0.1560$. If that raw second moment is incorrectly used as the variance estimate, the resulting standard error is $\sqrt{0.1560/5}=0.176636\ldots$, and the resulting statistic is $0.2000/0.176636\ldots=1.1323$. The correct centered calculation gives $1.3131$, not $1.1323$.

Common mistake avoided: calculating $n^{-1}\sum_i m_i^2$ and calling it a variance. Variance is calculated from deviations around the mean.

Worked Example: Keeping the Variance Convention Consistent

For four hypothetical claim-level comparisons, the fitted-model log-likelihood differences are $(-0.1,0.1,0.3,0.5)$. Calculate the mean, the $n$-denominator variance estimate, its standard error, and the standardized statistic. Also show what changes if the sample variance uses denominator $n-1$.

Method. The differences sum to $0.8$, so $\bar m=0.8/4=0.2000$. The deviations are $(-0.3,-0.1,0.1,0.3)$; their squares sum to $0.09+0.01+0.01+0.09=0.20$. With the asymptotic Vuong convention, $$ \hat\omega^2=\frac{0.20}{4}=0.0500,\qquad \widehat{\operatorname{SE}}(\bar m)=\sqrt{\frac{0.0500}{4}} =\sqrt{0.0125}=0.111803\ldots. $$ Thus $V=0.2000/0.111803\ldots=1.7889$.

If instead the variance of the individual differences is calculated with denominator $n-1=3$, it is $0.20/3=0.066667$. Using that convention consistently gives a standard error of $\sqrt{0.066667/4}=\sqrt{0.016667}=0.129099\ldots$, and a standardized value of $0.2000/0.129099\ldots=1.5492$. These are two finite-sample conventions, not two values to mix in one calculation. The requested asymptotic Vuong convention gives $1.7889$.

Check. The centered squared deviations sum to $0.20$, and dividing by four gives a smaller variance estimate than dividing by three, as expected. The standard error also uses a further division by $n=4$.

Common mistake avoided: switching denominators between the variance estimate and its stated convention without identifying the switch.

Worked Example: Starting From Paired Model Contributions

A hypothetical analyst fits two severity models to six claim amounts. For each claim, the following are the log-likelihood contributions under model 1 and model 2, respectively: $(-2.1,-2.3)$, $(-1.4,-1.3)$, $(-2.0,-2.4)$, $(-1.7,-1.8)$, $(-1.9,-1.7)$, and $(-1.2,-1.4)$. Calculate the Vuong mean difference, its estimated standard error, and the standardized statistic.

Method. Subtract the second contribution from the first in each pair. The six differences are $0.2,-0.1,0.4,0.1,-0.2,0.2$. Their sum is $0.6$, so $\bar m=0.6/6=0.1000$.

The deviations from $0.1000$ are $0.1,-0.2,0.3,0,-0.3,0.1$. Their squared values sum to $0.01+0.04+0.09+0+0.09+0.01=0.24$. Hence $$ \hat\omega^2=\frac{0.24}{6}=0.0400,\qquad \widehat{\operatorname{SE}}(\bar m)=\sqrt{\frac{0.0400}{6}} =\sqrt{0.0066667}=0.081650\ldots. $$ The standardized statistic is $V=0.1000/0.081650\ldots=1.2247$.

Check. The paired differences sum to $0.6$, and their centered squared deviations sum to $0.24$. Dividing the latter by six and then by six again before taking the square root gives the standard error of the mean, not the standard deviation of an individual difference.

Common mistake avoided: using the difference between total log likelihoods, $0.6$, as if it were enough to calculate a standard error. The paired contributions are needed to recover the variation across claims.

Conditions and Practical Checks

The variance calculation is most straightforward when observations are independent and identically distributed, the candidate likelihoods are properly specified for the observed data, and the log-likelihood differences have finite variance. In practice, the fitted parameters are estimated from the same observations, so the $m_i$ are not literally independent after fitting. Under the regularity conditions used for the Vuong comparison, the empirical variance of the fitted observation-level differences provides the relevant asymptotic variance estimate. The calculation does not by itself establish that those conditions hold.

Extra care is needed if observations are dependent, clustered, or weighted; if the models do not define comparable likelihoods on the same data; or if the log-likelihood contributions have very heavy tails. In such settings, the simple independent-observation standard error may not be appropriate. Likewise, if all observed differences are identical, $\hat\omega^2=0$ and the standard error is zero, so the standardized statistic is undefined (or cannot be treated as an ordinary finite statistic). Do not force the formula to produce a meaningful comparison in that case.

A reliable calculation can be audited with three checks: the differences must be paired observation by observation; the deviations used in the variance must sum to zero, apart from rounding; and the reported standard error must equal $\sqrt{\hat\omega^2/n}$ under the stated convention. If the raw second moment is used instead, check whether the omitted term $\bar m^2$ is nonzero.

Key takeaway: The Vuong standard error of the average log-likelihood difference is based on the centered spread of the individual differences: $\widehat{\operatorname{SE}}(\bar m)=\sqrt{\{n^{-1}\sum_i(m_i-\bar m)^2\}/n}$. Center first, use the stated variance convention consistently, and retain the paired contributions needed to measure variation across observations.

Check Your Understanding

For each problem, use the asymptotic Vuong convention $\hat\omega^2=n^{-1}\sum_i(m_i-\bar m)^2$, unless stated otherwise. Round standard errors and standardized statistics to four decimal places.

Four hypothetical observation-level differences are $(-0.3,0.1,0.2,0.4)$. Find $\bar m$, $\hat\omega^2$, the standard error of $\bar m$, and $V$. (Answer: $\bar m=0.1000$, $\hat\omega^2=0.0650$, standard error $=0.1275$, $V=0.7845$.)
Four paired claim-level differences are $(0.2,0.2,-0.2,-0.2)$. Calculate the centered variance and standard error. Explain numerically why using the raw second moment here happens not to change the variance estimate. (Answer: $\bar m=0$, centered variance $=0.0400$, standard error $=0.1000$; the raw second moment is also $0.0400$ because the mean is zero.)
Four differences from two fitted frequency models are $(-0.1,0.1,0.3,0.5)$. Calculate $V$ using the $n$-denominator convention, then recalculate using the $n-1$ sample-variance convention consistently. (Answer: $n$-denominator: $\bar m=0.2000$, standard error $=0.1118$, $V=1.7889$; $n-1$ convention: standard error $=0.1291$, standardized value $=1.5492$.)
Five differences are $(0.1,0.3,0.3,0.5,0.8)$. Find the mean, the centered variance, the standard error, and $V$. (Answer: $\bar m=0.4000$, $\hat\omega^2=0.0560$, standard error $=0.1058$, $V=3.7796$.)
Six paired log-likelihood contributions produce differences $(0.2,-0.1,0.4,0.1,-0.2,0.2)$. Calculate the raw second moment and verify the centered-variance identity, then calculate the standard error. (Answer: $\bar m=0.1000$, raw second moment $=0.0500$, centered variance $=0.0400=0.0500-(0.1000)^2$, standard error $=0.0816$.)

What You'll Learn