Why Pool Test Statistics?
Multiple imputation produces \(m\) completed data sets, and the same hypothesis can be tested separately in each one. Those tests will generally differ because the completed data sets differ. The D2 method combines the resulting test statistics into one approximate test of the common hypothesis, accounting for both their average size and their variation across imputations.
D2 is useful when the completed-data test statistics are available but the estimates and covariance matrices needed for pooling the test directly are not readily available. It is not a way to average p-values, and it does not produce a pooled coefficient estimate. Its input is a set of statistics for the same hypothesis, defined in the same way in every imputation.
Assumptions and Calculation
The \(D_i\) values must test the same null hypothesis, with the same \(k\) restrictions, in every completed data set. They should be nonnegative test statistics calibrated to an approximately chi-square distribution under that null. In particular, the test specification and parameterization must remain consistent across imputations. If a software output is a finite-denominator-df \(F\) statistic, it must not automatically be inserted as though it were a chi-square statistic; first establish that the statistic is on the scale required by the D2 procedure being used.
D2 uses the mean \(\bar D\) as the central signal and \(B\) as a measure of how much the test statistic changes across imputations. The factor \(1+1/m\) in \(r_2\) is part of the prescribed adjustment; omitting it changes the pooled result. The division by \(k(1+r_2)\) scales the average statistic for the number of restrictions and its imputation-related variation. The approximate denominator degrees of freedom reflect that the adjustment itself is estimated from a finite set of imputations. These formulas give an approximate reference distribution, not an exact finite-sample identity.
Confirm that every completed-data statistic tests the identical null using the same \(k\) restrictions and is on the appropriate chi-square scale.
Calculate \(\bar D\), then calculate \(B\) using denominator \(m-1\), not \(m\).
Calculate \(r_2\), \(F_{D2}\), and \(\nu_2\), retaining sufficient precision during intermediate calculations.
Obtain the upper-tail p-value from the \(F(k,\nu_2)\) distribution and state what it says about the common null hypothesis.
Worked Example: A Two-Restriction Test
Suppose three hypothetical completed data sets give Wald chi-square statistics \(40-\sqrt{6}\), \(40\), and \(40+\sqrt{6}\), for the same null hypothesis with \(k=2\) restrictions. These are nonnegative statistics on the required scale, and the null and degrees of freedom are unchanged across imputations.
Here \(m=3\), and the mean is
The deviations from the mean are \(-\sqrt{6}\), \(0\), and \(\sqrt{6}\). Therefore the between-imputation sample variance is
The adjustment is
Thus
The reference distribution is approximately \(F(2,76)\). For an \(F(2,\nu)\) variable, the upper-tail probability at \(f\) is \(\left[\nu/(\nu+2f)\right]^{\nu/2}\). The pooled p-value is therefore approximately \(\left[76/(76+2(16.6667))\right]^{38}=1.0\times10^{-6}\). At a conventional significance level of 0.05, this is strong evidence against the common null hypothesis. It is not the probability that the null hypothesis is true.
Between-Imputation Variation and Interpretation
If the \(D_i\) values are similar, \(B\) is small, so \(r_2\) is small and the adjusted statistic is close to \(\bar D/k\). When the statistics vary more across imputations, \(B\) and \(r_2\) increase. For a fixed \(\bar D\), the D2 adjustment then reduces \(F_{D2}\), reflecting uncertainty associated with the variation across completed data sets. The denominator degrees of freedom also depend on \(r_2\); report them with the test rather than reporting only the statistic.
D2 does not require every completed-data test to reach the same decision. A test statistic can be above a chosen cutoff in one completed data set and below it in another. The pooled conclusion comes from the D2 calculation, not from counting how many separate tests are significant. Likewise, averaging the individual p-values is not the D2 method: p-values are nonlinear transformations of the statistics and do not retain the required scale and variation information.
Worked Example: A One-Restriction Test With Modest Variation
Consider four hypothetical chi-square statistics for one common restriction: \(3.2,\ 3.6,\ 4.0,\ 4.4\). The test uses \(k=1\), and each statistic is on the chi-square scale under its null hypothesis.
Their mean is \(\bar D=(3.2+3.6+4.0+4.4)/4=3.8\). The deviations are \(-0.6,-0.2,0.2,0.6\), so
With \(m=4\), the variance adjustment is
Using the unrounded variance in the calculation gives \(F_{D2}\approx3.494\) and \(\nu_2\approx465.3\). The approximate \(F(1,465.3)\) upper-tail p-value is about 0.063. At \(\alpha=0.05\), the result does not provide sufficient evidence to reject the common null hypothesis. This conclusion is based on the pooled test, not on an informal comparison of the four separate statistics.
Diagnostics, Edge Cases, and Limitations
Before interpreting a D2 result, inspect the input statistics and the calculation. The procedure is suspect if statistics test different hypotheses, use different numbers of restrictions, or are reported on incompatible scales. Very large variation across imputations can yield a large adjustment and small denominator degrees of freedom; this is a reason to examine the imputation-specific analyses and the stability of the test, not to discard the adjustment. Also check that statistics have not been transcribed as p-values or as signed coefficients.
Rounding can materially affect \(B\), because it is computed from squared deviations. Keep the unrounded statistics available and round only the reported results. If only rounded inputs are available, calculate from those displayed inputs and state that choice; do not combine a variance calculated from rounded values with an adjustment claimed to use unavailable unrounded values.
If \(B=0\) and \(\bar D>0\), all imputation-specific statistics are identical, so \(r_2=0\). The limiting D2 statistic is \(F_{D2}=\bar D/k\), with an infinite denominator degrees of freedom; its reference distribution is the corresponding chi-square statistic divided by \(k\). If \(\bar D=0\), nonnegative inputs imply all \(D_i=0\); the test gives no evidence against the null, and the formula for \(r_2\) itself is undefined because it divides by zero.
D2 is an approximate test and its validity depends on the calibration of the input statistics and the quality of the imputations. It does not replace assessment of the missing-data assumptions or the imputation and analysis models. Compared with the D1 multivariate Wald test, D2 combines the test statistics rather than pooling the estimates and their covariance matrices to form the test. This can be convenient when compatible statistics are available, but it gives less direct information about pooled effect estimates and their directions. Likelihood-ratio-based pooling is another distinct approach; these methods should not be interchanged without checking their required inputs and assumptions.
Worked Example: More Between-Imputation Variation
Three hypothetical completed data sets produce statistics \(2,\ 5,\ 8\) for the same two-restriction chi-square test. The common null and \(k=2\) satisfy the comparability requirement, and the inputs are nonnegative statistics on the required scale.
The mean is \(\bar D=(2+5+8)/3=5\). The deviations are \(-3,0,3\), giving
Hence \(r_2=(1+1/3)(9/5)=2.4\), and
The approximate reference distribution is \(F(2,8.014)\); its upper-tail p-value is about 0.51. The pooled test does not reject the common null at the 0.05 level. Here the mean statistic is not especially large relative to \(k\), and the substantial variation across imputations further affects the adjustment and reference degrees of freedom.
Common Mistakes
- Averaging p-values. The D2 formula uses test statistics and their sample variance, not the arithmetic mean of their p-values.
- Using the wrong variance denominator. \(B\) is a sample variance with denominator \(m-1\), not \(m\).
- Omitting the finite-\(m\) factor. The adjustment uses \(1+1/m\); replacing it with 1 changes \(r_2\) and the pooled test.
- Mixing incompatible tests. A set of statistics is not poolable by D2 merely because each was labelled a test statistic. They must address the same null and have a compatible calibration.
- Rounding too early. Rounded deviations are squared in \(B\), so small input changes can propagate into \(r_2\), \(F_{D2}\), and \(\nu_2\). Preserve precision until the final report.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Four imputations give statistics \(4,4,4,4\) for the same test with \(k=2\). What are \(B\), \(r_2\), and the limiting D2 statistic?
- Why is the denominator of \(B\) equal to \(m-1\), rather than \(m\), in the D2 calculation?
- Can statistics for different null hypotheses be combined by the same D2 calculation just because they have the same \(k\)? Explain.
- Suppose \(B\) increases while \(\bar D\), \(m\), and \(k\) stay fixed. What happens to \(r_2\) and \(F_{D2}\)?
Solutions. (1) The mean is 4 and \(B=0\), since all deviations are zero. Thus \(r_2=0\), and \(F_{D2}\) is the limiting value \(\bar D/k=4/2=2\), referenced to \(F(2,\infty)\). (2) \(B\) is defined as the sample variance of the \(m\) statistics, so its denominator is \(m-1\); using \(m\) would not be the prescribed D2 calculation. (3) No. The restrictions must describe the same common null, not merely have the same count. (4) Since \(r_2=(1+1/m)B/\bar D\), increasing \(B\) increases \(r_2\). Since \(F_{D2}=\bar D/[k(1+r_2)]\), increasing \(r_2\) decreases \(F_{D2}\) when \(\bar D\) and \(k\) are fixed.