Why Averaging Test Statistics Is Not Rubin Pooling

Why a Test Statistic Is Not the Quantity Rubin Pools

After multiple imputation, each completed data set gives an estimate and a measure of its sampling variance. The estimates can differ because of ordinary sampling variation and because the missing values have been filled in differently. Rubin pooling combines both sources of uncertainty. A test statistic calculated separately in each completed data set is a different object: simply averaging those statistics does not, in general, reproduce the uncertainty-aware pooled test.

This distinction matters for Wald statistics, likelihood-ratio statistics, and p-values. They are computed nonlinearly from estimates, variances, or likelihoods. Averaging them does not automatically account for variation between imputations, and their usual single-data-set reference distributions do not automatically apply to their averages.

The factor \(1+1/m\) accounts for the fact that \(B\) is estimated using a finite number of imputations. The pooled standard error is \(\sqrt{T}\), not the average of the standard errors and not the standard error from one selected imputation. For a scalar test of \(H_0:q=q_0\), a common large-complete-sample reference is a \(t\) distribution with

Here \(\nu\) is the approximate degrees of freedom under the large-complete-sample form of Rubin’s rules. The p-value is calculated from this pooled \(t\) statistic and its reference distribution. When complete-data degrees of freedom are limited, an appropriate small-sample adjustment may be needed. In either case, the test is built from the pooled estimate and its total uncertainty—not by averaging imputation-specific test results.

Assumptions and the Scalar Wald Test

A Rubin-pooled scalar Wald test requires that the \(q_i\) represent the same estimand on the same scale in every imputation, and that each \(U_i\) estimates the sampling variance of that estimate under the analysis model. The imputation procedure must provide appropriate draws of the missing values, and the analysis and imputation models must support valid inference for the estimand. As with other analyses based on MICE, conclusions also depend on the missing-data and model assumptions; pooling does not repair a misspecified imputation or analysis model.

For a single completed data set, a Wald statistic is often formed by dividing an estimate’s distance from its null value by its estimated standard error. That statistic uses only that data set’s estimate and variance. Averaging such statistics across imputations neither pools the estimates nor combines their variance components according to Rubin’s rules. In particular, estimates that differ across imputations contribute to \(B\), which can increase \(T\) and change both the statistic and its degrees of freedom.

Why Averaging P-Values Does Not Fix the Problem

A p-value is a tail probability calculated from a test statistic under a specified null model and reference distribution. It is not an estimate of the parameter, a measure of uncertainty that can be averaged, or the probability that the null hypothesis is true. The p-values from separate imputations are conditional on their respective completed data sets; their arithmetic mean does not have a general pooled-test interpretation.

This example is not a rule that a pooled p-value must be larger or smaller than the average p-value. The difference depends on the estimates, their variances, and their variation across imputations. The point is that the average p-value has no general calibration as a pooled test.

Likelihood-Ratio and Joint Tests Need Their Own Pooling Method

Not every hypothesis is best tested using a scalar pooled Wald statistic. A likelihood-ratio test compares nested models: the restricted model imposes the null hypothesis, while the full model allows the tested parameters to vary. Its statistic and usual chi-square reference distribution are derived for one completed data set. Averaging likelihood-ratio statistics does not make their average follow that same chi-square distribution, nor does the average account for missing-data uncertainty in the required way.

For a joint hypothesis about several coefficients, a Wald approach should use pooled coefficient estimates and their pooled covariance information, with an appropriate pooled reference distribution. It should not average the separate imputation-specific joint Wald statistics. The earlier material on joint regression tests and the D3 method gives the relevant pooled procedures. The choice depends on the hypothesis and the model: use a pooled Wald procedure when the target is naturally expressed through pooled estimates and covariance, and a pooled likelihood-ratio procedure such as D3 when the test is based on nested-model likelihoods.

Practical Checks and Common Pitfalls

Before reporting a test, verify that the same parameter, contrast, model specification, and null value were used in every imputation. If the parameter’s meaning or scale changes between imputations, the pooled result no longer targets a single clearly defined quantity. Also inspect the estimates and within-imputation variances: extreme values, implausible model fits, or large differences can signal instability or model problems. Such diagnostics do not replace the pooling calculation, but they can reveal when its inputs are questionable.

• Wrong reasoning: “Each completed-data Wald test uses the correct formula, so their average is the pooled Wald test.” Correction: Rubin pooling combines \(q_i\) and \(U_i\), then incorporates \(B\) into \(T\) and uses an appropriate pooled reference distribution.
• Wrong reasoning: “Averaging likelihood-ratio statistics and comparing the mean with a chi-square distribution is valid because each statistic is chi-square under the null.” Correction: The distribution of the average is not thereby the single-test chi-square distribution; use a pooled likelihood-ratio method such as D3.
• Wrong reasoning: “The mean p-value summarizes the evidence across imputations.” Correction: A mean of p-values is not generally a calibrated pooled p-value. Pool the estimates and variances for a scalar Wald test, or use a method designed for the chosen hypothesis.

No pooling method can compensate for invalid input. Inferences may be unreliable if the imputation or analysis model is misspecified, if the estimand changes across imputations, or if the assumptions needed for the chosen reference distribution are poor. Rubin pooling addresses the combination of estimates and uncertainty; it does not by itself validate the scientific model or missing-data assumptions.