The D3 Method For Pooling Likelihood-Ratio Tests

Why Pool Likelihood-Ratio Evidence?

A likelihood-ratio test compares a model that allows the parameter of interest to vary with a nested model that imposes the null hypothesis. With multiply imputed data, this comparison can be made separately in each completed data set, but the resulting likelihood-ratio statistics differ across imputations. The D3 method combines the evidence while accounting for this between-imputation variation. It is useful when the likelihoods from both fitted models are available and the null hypothesis is naturally expressed as restrictions on model parameters.

D3 is not obtained by simply averaging the likelihood-ratio statistics and referring that average to a chi-square distribution. Its pooled statistic also uses a likelihood-based combination of the imputation-specific results, and its reference degrees of freedom reflect the variation across imputations.

Models and Assumptions

Definition: For each imputed data set \(j=1,\ldots,m\), fit an unrestricted model and a nested model constrained by the null hypothesis. Let \(\ell_{1j}\) be the maximized log-likelihood for the unrestricted model and \(\ell_{0j}\) the maximized log-likelihood under the null constraints. The imputation-specific likelihood-ratio statistic is \(D_j=-2(\ell_{0j}-\ell_{1j})\). If the null imposes \(k\) independent restrictions, \(k\) is the numerator degrees of freedom for the pooled test.

The models must use the same outcome, observations, likelihood definition, and covariate structure apart from the restrictions being tested. They must be genuinely nested: the constrained model is obtained by imposing the null restrictions on the unrestricted model. Under regularity conditions for a likelihood-ratio test, the complete-data statistic has an approximate chi-square distribution with \(k\) degrees of freedom under the null. This calibration is an ingredient in the D3 approximation, not a guarantee that the approximation is accurate in every sample.

The imputed data should be suitable for the intended analysis: in particular, the imputation procedure should reflect uncertainty about missing values and be compatible with the analysis model. Each model must converge in every imputation, and the same \(k\) restrictions must be tested each time. A failed fit, non-nested models, or changing analysis samples undermine the pooled comparison.

The D3 Calculation

Calculate the mean of the imputation-specific likelihood-ratio statistics, \(\bar D\), and the mean relative likelihood under the constrained models. The latter is transformed back to a likelihood-ratio scale to obtain \(\tilde D\). Then calculate the relative increase in the test statistic due to imputation, \(r_3\), the D3 statistic, and its approximate denominator degrees of freedom:

\bar D=\frac{1}{m}\sum_{j=1}^{m}D_j,\qquad \tilde D=-2\log\left\{\frac{1}{m}\sum_{j=1}^{m}\exp(-D_j/2)\right\},

r_3=\left(1+\frac{1}{m}\right)\frac{\bar D-\tilde D}{\tilde D},\qquad F_3=\frac{\tilde D}{k(1+r_3)},\qquad \nu_3=4+(km-4)\left[1+\frac{1-2/(km)}{r_3}\right]^2.

The approximate reference distribution is \(F(k,\nu_3)\); the p-value is the upper-tail probability \(P(F_{k,\nu_3}\geq F_3)\). The factor \(1+r_3\) reduces the test statistic when the imputation-specific evidence is dispersed. The value \(\tilde D\) is based on averaging relative likelihoods, not on taking the arithmetic average of the \(D_j\)'s.

Why the Pooled Relative Likelihood Is No Larger Than the Mean

For nested models, each \(D_j\) is nonnegative. The function \(g(d)=\exp(-d/2)\) is convex, since \(g''(d)=\exp(-d/2)/4>0\). Jensen's inequality therefore gives:

\frac{1}{m}\sum_{j=1}^{m}\exp(-D_j/2) \geq \exp\left(-\frac{1}{2m}\sum_{j=1}^{m}D_j\right) =\exp(-\bar D/2).

Taking negative twice the logarithm reverses the comparison because the logarithm is increasing and is multiplied by \(-2\). Thus \(\tilde D\leq\bar D\). When \(\tilde D>0\), this implies \(r_3\geq0\). Equality holds when all the \(D_j\)'s are identical. This also provides a basic numerical check: a calculation yielding \(\tilde D>\bar D\) or a negative \(r_3\) for valid nested-model statistics should be investigated.

Worked Example: Setting Up the Required Nested Models

Suppose a hypothetical logistic-regression analysis tests whether two treatment indicators contribute jointly to the log odds of an outcome, adjusting for age and site. The unrestricted model contains both treatment coefficients; the null hypothesis is that both are zero. In each completed data set, the unrestricted model includes age, site, and both treatment indicators. The constrained model includes age and site but sets the two treatment coefficients to zero. The models are nested and differ by \(k=2\) restrictions, provided the same records and likelihood specification are used in both fits.

In one hypothetical imputation, suppose the maximized log-likelihoods are \(\ell_{1j}=-102.4\) and \(\ell_{0j}=-104.1\). Then \(D_j=-2[-104.1-(-102.4)]=-2(-1.7)=3.4\). The constrained model has the lower maximized log-likelihood, as expected for nested models. The value 3.4 is one imputation-specific contribution, not the pooled test statistic. The same two models must be fitted in every imputation before applying D3.

Worked Example: Calculating and Interpreting a D3 Test

Consider five hypothetical imputations for a test with \(k=2\) restrictions. Suppose their likelihood-ratio statistics are \(0.060,\ 0.060,\ 2.520,\ 2.520,\ 2.540\). They are nonnegative, consistent with nested models. Their mean is \(\bar D=(0.060+0.060+2.520+2.520+2.540)/5=7.700/5=1.540\).

Next calculate the mean relative likelihood: \[ \frac{\exp(-0.060/2)+\exp(-0.060/2)+\exp(-2.520/2)+\exp(-2.520/2)+\exp(-2.540/2)}{5} \] \[ =\frac{2(0.97045)+2(0.28365)+0.28083}{5} \approx 0.5578. \] Thus \(\tilde D=-2\log(0.5578)\approx1.168\). Using the unrounded relative-likelihood calculation gives approximately \(1.1675\); the following results use that unrounded value.

With \(m=5\), \(r_3=(1+1/5)(1.540-1.1675)/1.1675\approx1.2(0.3725/1.1675)\approx0.3829\). The D3 statistic is \(F_3=1.1675/[2(1+0.3829)]\approx0.4221\). Here \(km=10\), so \[ \nu_3=4+(10-4)\left[1+\frac{1-2/10}{0.3829}\right]^2 =4+6\left(1+\frac{0.8}{0.3829}\right)^2 \approx61.28. \]

The approximate reference distribution is \(F(2,61.28)\). Its upper-tail probability at \(0.4221\) is approximately \(0.657\). At a significance level of \(\alpha=0.05\), this is not evidence against the joint null hypothesis. The result does not establish that the treatment coefficients are exactly zero; it says that these completed-data likelihood comparisons do not provide strong evidence against their joint null under the stated models and assumptions.

Diagnostics, Edge Cases, and Alternatives

Inspect the \(D_j\)'s before trusting the pooled result. Large variation, a single extreme statistic, or substantial differences in convergence behavior can make the approximation sensitive to particular imputations. Check that the constrained log-likelihood never exceeds its unrestricted counterpart beyond negligible numerical error. Verify that each fit used the intended records and restrictions, and compute the pooled relative likelihood using adequate numerical precision. For very large \(D_j\), direct exponentiation can underflow; a log-sum-exp calculation is safer.

The denominator degrees-of-freedom formula requires care when \(r_3\) is zero or very small. If \(r_3>0\) and \(km>4\), its limit as \(r_3\) tends to zero from above is \(\nu_3\to\infty\). This does not justify applying the formula uncritically when \(km\leq4\): for \(km=4\), the expression is 4 for positive \(r_3\), while for \(km<4\) it can fail to provide meaningful degrees of freedom. If all \(D_j\)'s are identical, then \(r_3=0\) and the displayed formula has a zero denominator. Treat that case separately rather than inserting zero; an asymptotic chi-square calibration may be considered only when its regularity and sample-size conditions are defensible. With very few imputations or sparse data, the finite-sample approximation can be particularly fragile.

D1 combines pooled coefficient estimates and their covariance information in a multivariate Wald test. D2 pools imputation-specific test statistics. D3 instead requires fitting both the unrestricted and null-constrained models in every imputation and uses the likelihood-ratio information. These methods are not guaranteed to give identical results in finite samples. D3 can be attractive when the nested likelihoods are readily available, but it may be burdensome for complex models and depends on successful, comparable fits. It is not a substitute for checking model specification or imputation quality.

Common pitfall: Averaging the \(D_j\)'s and comparing \(\bar D\) directly with a chi-square distribution ignores the D3 adjustment for variation across imputations. Calculate \(\tilde D\), \(r_3\), \(F_3\), and the applicable denominator degrees of freedom; if the degrees-of-freedom formula is inapplicable, do not force a numerical result from it.

Worked Example: Detecting an Invalid D3 Calculation

Suppose an analyst has \(m=2\) imputations and tests \(k=1\) restriction. Then \(km=2\), so \(1-2/(km)=0\) and the bracket in the displayed degrees-of-freedom expression is 1 for every positive \(r_3\). Thus \(\nu_3=4+(2-4)(1)^2=2\). Whether the approximation is suitable or justified with so few imputations is a separate concern; the analyst should state this limitation and consider an appropriate alternative pooling procedure or a more defensible analysis design.

Key takeaway: D3 pools likelihood-ratio evidence using both the average statistic and the mean relative likelihood. Its use depends on comparable nested-model fits across imputations, and its approximate F reference distribution must be checked rather than assumed.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a D3 test of three independent null restrictions, what are \(k\) and the numerator degrees of freedom?
Why should the constrained model's maximized log-likelihood not exceed the unrestricted model's in a valid nested comparison?
If all imputation-specific \(D_j\)'s are equal, what happens to \(r_3\), and why can the finite-df formula not simply be evaluated by substituting zero?
What is the main distinction between D3 and pooling a mean likelihood-ratio statistic against a chi-square reference distribution?

Solutions. (1) \(k=3\), so the numerator degrees of freedom are 3. (2) The unrestricted model contains all parameter values allowed by the constrained model, so maximizing over the larger parameter space cannot produce a lower maximized likelihood; consequently \(\ell_{0j}\leq\ell_{1j}\) and \(D_j\geq0\). (3) If all the \(D_j\)'s are identical and positive, Jensen's inequality is an equality, so \(\tilde D=\bar D\) and \(r_3=0\). If they are all zero, then \(\tilde D=\bar D=0\) and \(r_3\) is undefined because its denominator is zero. The degrees-of-freedom expression divides by \(r_3\), so it is undefined at zero; moreover, its limit must not be assumed for every \(km\). (4) D3 uses the mean relative likelihood to obtain \(\tilde D\), adjusts for between-imputation variation through \(r_3\), and uses an approximate F reference distribution rather than treating the arithmetic mean alone as a chi-square statistic.