Why Model Overlap Matters
A Vuong comparison uses observation-level log-likelihood differences to compare two fitted models. Its usual standard-normal reference is not automatic: it depends on the candidate models being distinguishable in the relevant sense. If the models can generate the same distribution, their overlap can make the variance used to standardize the statistic zero, or can change the statistic’s limiting behavior.
The key exam skill is to separate three questions: Are the candidate model families disjoint? Can they generate a common distribution? And, at the parameter values relevant to the comparison, do they give the same distribution for the observed data? These questions are related, but they are not interchangeable.
The definition is about probability distributions, not about whether two density formulas look different or whether two parameter names match. For example, two differently written densities can describe exactly the same law at particular parameter values. Conversely, two models may overlap at some parameters but give different densities at other parameters.
The Ordinary Vuong Statistic and Its Condition
For independent, identically distributed observations \(X_1,\ldots,X_n\), fit both models by maximum likelihood and define the observation-level log-likelihood difference, with the direction stated explicitly, by
When the models are distinguishable in the relevant comparison, the usual Vuong statistic is \(V=\sqrt{n}\bar d/\hat{\omega}\). A positive value favors \(M_1\) in average fitted log score; a negative value favors \(M_2\). The standard-normal reference concerns the null hypothesis of equal expected log scores, subject to the regularity conditions for the comparison. It is not a general rule for every pair of models.
The essential variance condition is that the population log-likelihood difference has positive variance. If the two candidate densities agree almost everywhere at the relevant parameter values, then their log-density difference is zero almost surely, and its variance is zero. Division by an estimate of a zero variance cannot produce a standard-normal statistic.
Why Positive Variance Leads to a Standardized Comparison
The justification begins with the population log-likelihood difference \(D=\log f_1(X;\theta_1)-\log f_2(X;\theta_2)\) at the probability limits of the fitted parameters. Under the equal-expected-log-score null, \(E[D]=0\). If \(D\) has finite, positive variance \(\omega^2\), then the central limit theorem gives the following result:
Replacing the population standard deviation \(\omega\) with a consistent estimate gives the familiar standardized statistic, under the relevant regularity conditions. The positivity requirement matters: the central limit theorem does not turn a random variable that is identically zero into a nondegenerate normal variable. At observational equivalence, \(D=0\) almost surely and \(\omega^2=0\); the argument for the ordinary reference distribution breaks down.
There is a second caution. In overlapping models, parameters may be estimated under restrictions or may converge to a common distribution under the null. Parameter estimation can then affect the asymptotic behavior of the likelihood comparison. One cannot infer the appropriate reference distribution merely by calculating a sample standard deviation and obtaining a nonzero number.
Worked Examples
Worked Example: A Distinguishable Bernoulli Comparison
A hypothetical audit records whether a randomly selected policy file contains a particular documentation error. Compare two fixed models: \(M_1\) assigns probability \(0.30\) to an error, and \(M_2\) assigns probability \(0.60\). In a hypothetical sample of six independent files, three contain the error and three do not. Calculate the fitted-direction log-score statistic using these fixed probabilities, and assess whether the models overlap.
Method. Treat an error as \(X_i=1\) and no error as \(X_i=0\). For a success, the log-likelihood difference in the direction \(M_1-M_2\) is \(\log(0.30/0.60)=\log(0.5)=-0.693147\). For a failure, it is \(\log(0.70/0.40)=\log(1.75)=0.559616\).
There are three observations of each type, so the sum of the six differences is \(3(-0.693147)+3(0.559616)=-2.079441+1.678848=-0.400593\). Therefore, \(\bar d=-0.400593/6=-0.066766\).
The deviations from the mean are \(-0.693147-(-0.066766)=-0.626381\) for each error and \(0.559616-(-0.066766)=0.626382\) for each non-error, up to rounding. Thus \(\hat{\omega}^2=[6(0.626381)^2]/6=0.3924\), and \(\hat{\omega}=0.6264\). The standardized value is \(V=\sqrt{6}(-0.066766)/0.6264=-0.261\), rounded to three decimals.
Classification and check. The first model can generate only a Bernoulli distribution with probability \(0.30\), while the second can generate only one with probability \(0.60\). These are different distributions, so the two singleton model families are distinguishable. The negative sign agrees with the observed total log-likelihood difference of \(-0.400593\): this sample favors \(M_2\) by log score. Since \(n=6\) is small, the statistic is an arithmetic illustration, not a claim that the asymptotic reference is accurate for this sample. The common mistake is to treat a standard-normal value as automatically reliable without checking the model relationship and sample conditions.
Worked Example: An Exponential Distribution Inside a Weibull Model
A hypothetical severity model compares an exponential distribution with scale \(\theta=1\) to a Weibull distribution with shape \(k=1\) and scale \(\theta=1\). For three hypothetical losses \(x=0.5,1.0,1.5\), determine whether this parameter pair is observationally equivalent and whether the ordinary Vuong statistic can be calculated from the observation-level differences.
Method. The exponential density with scale \(\theta\) is \(f_E(x;\theta)=\theta^{-1}e^{-x/\theta}\), for \(x>0\). The Weibull density is \(f_W(x;k,\theta)=(k/\theta)(x/\theta)^{k-1}e^{-(x/\theta)^k}\), for \(x>0\). At \(k=1\), the Weibull density becomes \((1/\theta)e^{-x/\theta}\), exactly the exponential density with the same scale.
At \(\theta=1\), both densities at \(x=0.5\) are \(e^{-0.5}=0.606531\); at \(x=1.0\), both are \(e^{-1}=0.367879\); and at \(x=1.5\), both are \(e^{-1.5}=0.223130\). Each observation-level difference is therefore \(d_i=\log f_E(x_i;1)-\log f_W(x_i;1,1)=0\).
The total log likelihood in either model is \(\log L=-0.5-1.0-1.5=-3.000\), so \(L=e^{-3}=0.049787\). The mean difference is \(\bar d=0\), and the estimated variance is \(\hat{\omega}^2=(0^2+0^2+0^2)/3=0\). Consequently \(V=\sqrt{3}(0)/0\) is undefined, not zero.
Classification and check. The exponential family is contained in the Weibull family because every exponential scale corresponds to a Weibull distribution with \(k=1\). The model families overlap, and the specified distributions are observationally equivalent. The common mistake is to report \(V=0\) just because the numerator is zero. A zero numerator divided by a zero denominator is undefined; the data provide no log-score distinction at this parameter pair.
Worked Example: Overlap Does Not Make the Nested Likelihood-Ratio Test Invalid
Suppose three hypothetical, independent measurements of a standardized payment index are \(1,1,2\). Compare \(M_1:N(0,1)\), where both mean and variance are fixed, with \(M_2:N(\mu,1)\), where the variance remains fixed at \(1\) and the mean is estimated. Calculate the plug-in Vuong quantity and the likelihood-ratio statistic, then identify the appropriate comparison.
Method. The sample mean is \(\bar x=(1+1+2)/3=4/3\), so the maximum-likelihood estimate under \(M_2\) is \(\hat{\mu}=4/3\). The models overlap at \(\mu=0\), where \(M_2\) reproduces \(M_1\); in fact, \(M_1\) is nested within \(M_2\).
For one observation \(x\), the log-density difference in the direction \(M_1-M_2\) is \(\log f_1(x)-\log f_2(x;\hat{\mu})=[(x-\hat{\mu})^2-x^2]/2=\hat{\mu}^2/2-\hat{\mu}x\). Since \(\hat{\mu}^2/2=8/9\), the differences for the observations \(1,1,2\) are \(8/9-4/3=-4/9\), \(-4/9\), and \(8/9-8/3=-16/9\). Their mean is \(\bar d=[-4/9-4/9-16/9]/3=-8/9\).
Using the Vuong variance convention with divisor \(n=3\), the deviations from the mean are \(4/9,4/9,-8/9\). Hence \(\hat{\omega}^2=[16/81+16/81+64/81]/3=32/81\), and \(\hat{\omega}=4\sqrt{2}/9\). The plug-in standardized quantity is \(V=\sqrt{3}(-8/9)/(4\sqrt{2}/9)=-2\sqrt{3/2}\approx-2.449\).
This value should not be assigned the ordinary distinguishable-model Vuong calibration. The candidates overlap, and the fitted mean is estimated in the larger model. Instead, the likelihood-ratio statistic is \(2(\ell_2-\ell_1)=-2\sum_i d_i=-2(-8/3)=16/3=5.333\). Under the null \(\mu=0\), the normal model with known variance satisfies the regular nested-model conditions; the likelihood-ratio statistic has a \(\chi^2\) distribution with one degree of freedom (here, this result is exact). Its upper-tail probability is about \(0.0209\).
Check and trap. The larger model has a likelihood advantage of \(8/3\), so twice that advantage is \(16/3\), as calculated. The trap is to call the plug-in value \(-2.449\) an ordinary Vuong test result simply because its sample variance is positive. For this nested comparison, use the likelihood-ratio procedure with its appropriate null and conditions.
What to Do When Models Overlap
Overlap is a warning to check the comparison’s structure before choosing a reference distribution. It is not, by itself, proof that every possible comparison between the models is unusable. The key issue is whether the null and fitted distributions put the log-likelihood differences in a regime where the ordinary positive-variance argument applies.
| Relationship | What it implies | Practical consequence |
|---|---|---|
| Distinguishable families | No distribution belongs to both families. | The ordinary Vuong framework may apply if its other regularity conditions, including positive variance, hold. |
| Overlapping families | At least one distribution belongs to both, but the families are not identical. | Check whether the comparison is at or near the intersection; do not assume the ordinary reference automatically applies. |
| Observationally equivalent parameter values | The candidate densities are equal almost everywhere at those values. | The log-density differences have zero population variance there; the ordinary statistic may be undefined or have nonstandard behavior. |
If one model is nested in another and the null is a regular parameter restriction, a likelihood-ratio test with its appropriate degrees of freedom is often the natural comparison. This conclusion depends on the usual likelihood-ratio conditions; boundary parameters, unidentified nuisance parameters, or other nonregular features can change the calibration. For overlapping models that are not nested in a regular way, use a comparison method whose null distribution has been justified for that model relationship. There is no universal standard-normal fallback.
Common Exam Traps
- Classifying by formula rather than distribution. Different density expressions can describe the same law at specific parameter values. Check equality of distributions, not notation.
- Confusing overlap with equivalence. Overlap means there is at least one common distribution; it does not mean the two families are identical. Equivalence at a parameter pair is a statement about that pair’s observable distribution.
- Calling an undefined statistic zero. If both the mean difference and its estimated standard deviation are zero, the ratio is undefined.
- Using a nonzero sample variance as a complete validity check. Estimated parameters and an overlapping null can affect the asymptotic distribution even when the realized sample variance is positive.
- Ignoring the sign convention. With \(d_i=\log f_1-\log f_2\), positive average differences favor \(M_1\), and negative differences favor \(M_2\).
Check Your Understanding
For each problem, classify the model relationship or choose the appropriate conclusion. Answers are provided for checking.
- Model \(M_1\) contains only a Bernoulli distribution with success probability \(0.25\), and \(M_2\) contains only one with success probability \(0.70\). Are the families distinguishable or overlapping? (Answer: Distinguishable; the two singleton families contain different distributions.)
- An exponential model has unknown scale \(\theta>0\), and a Weibull model has unknown scale \(\theta>0\) and fixed shape \(k=1\). Classify the families. (Answer: They are observationally equivalent families; both contain exactly the exponential distributions with positive scale.)
- At a particular fitted parameter pair, \(f_1(X)=f_2(X)\) almost surely. What are the population mean and variance of the log-density difference, and is the ordinary standardized Vuong statistic defined at that pair? (Answer: Mean \(0\), variance \(0\); the statistic is undefined because its denominator is zero.)
- A model \(M_1:N(0,\sigma^2)\), with unknown \(\sigma^2>0\), is compared with \(M_2:N(\mu,\sigma^2)\), with unknown \(\mu\) and \(\sigma^2>0\). Identify the relationship and a natural test for the regular null that \(\mu=0\). (Answer: \(M_1\) is nested in \(M_2\); use the one-degree-of-freedom nested likelihood-ratio test when its regularity conditions hold.)
- A candidate comparison has overlapping model families, but the fitted observation-level differences are not all equal and their estimated sample variance is positive. Does that alone establish the ordinary standard-normal Vuong calibration? (Answer: No; positive sample variance alone does not establish the required asymptotic conditions for an overlapping-model null.)