Pooled Scalar Tests And P-Values

Testing a Scalar Hypothesis After Multiple Imputation

After multiple imputation, a scalar parameter has one estimate from each completed data set. A test of that parameter should reflect both the uncertainty within completed data sets and the variation in estimates across imputations. Testing each completed-data estimate separately and combining the resulting p-values does not do this. The scalar pooling approach instead tests the pooled estimate using its pooled standard error and a $t$ reference distribution with degrees of freedom that account for imputation uncertainty.

Let $Q_j$ be the estimate of the same scalar estimand from imputation $j$, and let $U_j$ be its estimated variance, for $j=1,\ldots,m$. Write $\bar{Q}$ for the mean of the $m$ estimates, $\bar{U}$ for the mean of their estimated variances, $B$ for the sample variance of the estimates, and $T$ for Rubin’s total variance. The earlier material on Rubin’s Rules defines these pooled quantities; here they are used to form and interpret a hypothesis test.

Definition: To test the scalar null hypothesis $H_0: Q=Q_0$, where $Q_0$ is a specified value, calculate $$ T=\bar{U}+\left(1+\frac{1}{m}\right)B,\qquad SE(\bar{Q})=\sqrt{T},\qquad t=\frac{\bar{Q}-Q_0}{\sqrt{T}}. $$ For a two-sided alternative $H_a: Q\ne Q_0$, the pooled p-value is $$ p=2P\left(t_{\nu}\geq |t|\right), $$ where $t_{\nu}$ denotes a random variable with a Student $t$ distribution and $\nu$ is the degrees of freedom appropriate for the pooled analysis.

The $t$ reference distribution is generally an approximation, not a claim that the test statistic has an exact $t$ distribution in every imputation setting. Its degrees of freedom reflect the finite number of imputations and, when relevant, the finite complete-data degrees of freedom. Use the degrees of freedom appropriate to the analysis: Rubin’s large-sample degrees of freedom are commonly used when complete-data degrees of freedom are effectively unlimited; when those degrees of freedom are limited, the Barnard–Rubin small-sample adjustment may be appropriate. Those choices affect the tail area, so the selected $\nu$ must be used consistently in calculating the p-value and any corresponding confidence interval.

Why the Pooled Statistic Has This Form

The numerator measures the distance between the pooled estimate and the value asserted by the null hypothesis. The denominator measures the uncertainty of that pooled estimate, including both the average within-imputation variance and the additional component due to differences between imputations. Dividing the distance by this standard error expresses the discrepancy in standard-error units.

Under $H_0$, and when the imputations and variance estimates support the usual multiple-imputation approximations, the standardized discrepancy is compared with a $t$ distribution using the pooled degrees of freedom. For a two-sided test, discrepancies in either direction count as evidence against the null. Symmetry of the $t$ distribution gives the factor of two in the p-value formula.

P\left(|t_{\nu}|\geq |t_{\mathrm{obs}}|\right) =P\left(t_{\nu}\geq |t_{\mathrm{obs}}|\right) +P\left(t_{\nu}\leq -|t_{\mathrm{obs}}|\right) =2P\left(t_{\nu}\geq |t_{\mathrm{obs}}|\right).

Thus the p-value is the probability, calculated under the null model and the chosen reference distribution, of obtaining a test statistic at least as far from zero as the observed statistic. It is not the probability that the null hypothesis is true, nor is it the probability that the observed data arose “by chance.”

Worked Pooled Tests

Worked Example: Testing a Nonzero Null Value

Suppose six hypothetical imputations estimate the same mean difference in a clinical outcome, measured in points. The pooled estimate is $\bar{Q}=1.200$ points, the pooled within-imputation variance is $\bar{U}=0.0100$ points squared, and the between-imputation variance is $B=0.00200$ points squared. Test $H_0:Q=1.500$ against $H_a:Q\ne1.500$, using Rubin’s large-sample degrees of freedom.

First calculate the total variance:

T=0.0100+\left(1+\frac{1}{6}\right)(0.00200) =0.0100+0.0023333 =0.0123333\ \text{points}^2.

The standard error is $\sqrt{0.0123333}=0.111055$ points. The pooled test statistic is

t=\frac{1.200-1.500}{0.111055}=-2.7014.

For these inputs, the large-sample degrees of freedom are

\nu=(m-1)\left(1+\frac{\bar{U}}{(1+1/m)B}\right)^2 =5\left(1+\frac{0.0100}{(7/6)(0.00200)}\right)^2 \approx139.69.

The two-sided p-value is $2P(t_{139.69}\geq2.7014)\approx0.007761$. At significance level $\alpha=0.05$, this is evidence against a mean difference of 1.500 points. The result is an association-level inference about the specified estimand; it does not by itself establish a causal effect.

Worked Example: A Test Statistic Near the Significance Threshold

In a second hypothetical analysis, suppose the pooled estimate of a treatment-group contrast is $\bar{Q}=2.400$ score units, with $T=0.0520$ score units squared and $\nu=75.1$. Test $H_0:Q=2.000$ against a two-sided alternative.

The standard error is $\sqrt{0.0520}=0.2280$ score units. Therefore,

t=\frac{2.400-2.000}{0.2280}=1.754.

Using a $t$ distribution with 75.1 degrees of freedom gives a two-sided p-value of approximately $0.0836$. At $\alpha=0.05$, the result is not statistically significant. This does not show that the null value is correct: it says that the observed standardized discrepancy is not sufficiently unusual under this test’s 5% decision rule. The point estimate is 0.400 score units above the null value, but its uncertainty is substantial relative to that difference.

Worked Example: How Degrees of Freedom Affect the Conclusion

Suppose a hypothetical pooled analysis reports $\bar{Q}=12.4$ milligrams per litre, $Q_0=12.0$ milligrams per litre, $SE(\bar{Q})=0.200$ milligrams per litre, and an appropriate small-sample pooled degrees of freedom of $\nu=8$. The test statistic is $t=(12.4-12.0)/0.200=2.00$. With 8 degrees of freedom, the two-sided p-value is approximately $0.0805$, so the test does not reject at $\alpha=0.05$.

If someone replaced the $t$ reference distribution with the standard normal distribution, the two-sided tail area for a statistic of 2.00 would be approximately $0.0455$. That would change the decision. The discrepancy occurs because a $t$ distribution with 8 degrees of freedom has heavier tails than the standard normal distribution. Using the normal approximation when the pooled degrees of freedom are small can therefore understate the p-value.

Conditions, Diagnostics, and Alternatives

A pooled scalar test is appropriate when all $m$ estimates target the same scalar quantity on the same scale, the corresponding $U_j$ values estimate the variance of those estimates, and the imputation procedure yields imputations suitable for the intended inference. The conventional test also relies on the usual approximate sampling distribution for the estimates and on a total variance that can be estimated meaningfully. Its p-value is an approximation whose quality can be affected by poor imputation-model behavior, too few imputations, or severe non-normality of the estimator.

Check that the estimate and null value are expressed in the same units and parameter scale, that $T$ is positive, and that the degrees of freedom match the analysis. Examine whether estimates across imputations appear unusually unstable or whether a small number of imputations is driving the result. Sensitivity of the pooled estimate or p-value to increasing $m$ can indicate excessive Monte Carlo error. These checks do not replace scrutiny of the imputation and analysis models; a neatly calculated test cannot repair a poorly specified model.

The main alternatives answer different questions or rely on different approximations:

Approach	What it does	Why it may not be suitable here
Pooled scalar $t$ test	Tests the pooled estimate using total variance and pooled degrees of freedom.	It is the relevant approach for the scalar null described here, subject to its assumptions.
Standard normal test	Uses normal rather than $t$ tail probabilities.	It can be a reasonable approximation with very large degrees of freedom, but can give misleading tail areas when degrees of freedom are small.
Separate tests in each imputation	Produces one test statistic or p-value per completed data set.	Those individual tests do not account for between-imputation uncertainty as a pooled scalar test does; their p-values should not simply be averaged.
Single completed-data analysis	Uses only one imputed data set for inference.	It discards the variation across imputations and does not represent the full imputation uncertainty.

This method is for a scalar null hypothesis. A collection of coefficient restrictions is a joint-testing problem, not several independent scalar tests that can automatically be treated as one joint test. Also, if the parameter or its variance is at a boundary or nearly degenerate, the ordinary scalar $t$-test approximation may be unreliable; such cases require separate care rather than mechanically applying the formula above.

Common Mistakes

Averaging imputation-specific p-values. This does not reproduce the pooled estimate, total variance, or pooled reference distribution. Calculate one scalar test from the pooled quantities instead.
Using only $\bar{U}$ in the denominator. The total variance also includes $(1+1/m)B$. Omitting it ignores between-imputation uncertainty and can make the standard error too small.
Using a normal tail area automatically. Use the selected pooled degrees of freedom and its $t$ distribution; a normal approximation is not interchangeable with a small-$\nu$ test.
Calling a nonsignificant result proof of no effect. A large p-value is not evidence that the null is true. Report the estimate, its uncertainty, and the test result in context.
Reporting a p-value without its direction or context. The two-sided p-value uses $|t|$, but the sign of $t$ still indicates whether the pooled estimate is above or below the null value.

Key takeaway: For a scalar null, subtract the null value from the pooled estimate, divide by the square root of Rubin’s total variance, and evaluate the absolute statistic against the appropriate pooled $t$ distribution. The p-value and any confidence interval should use consistent variance and degrees-of-freedom choices.

Check Your Understanding

Answer each question, then compare your reasoning with the short solutions below.

A pooled analysis gives $\bar{Q}=5.0$, $Q_0=4.5$, and $SE(\bar{Q})=0.25$. What is the test statistic, and in which direction is the estimate from the null?
Why is the two-sided p-value based on the absolute value of the test statistic?
Suppose $\bar{U}=0.020$, $B=0.006$, and $m=4$. Calculate $T$ and $SE(\bar{Q})$.
A researcher averages the p-values from five imputed data sets. Explain the problem with this procedure.

Solutions. (1) $t=(5.0-4.5)/0.25=2.00$; the estimate is above the null. (2) A two-sided alternative counts departures above and below the null, so the tail areas beyond $+|t|$ and $-|t|$ are added. (3) $T=0.020+(1+1/4)(0.006)=0.0275$, so $SE(\bar{Q})=\sqrt{0.0275}=0.1658$. (4) The average does not use the pooled total variance or the pooled degrees of freedom and therefore is not the scalar pooled test; calculate the statistic and p-value from the pooled estimate and its uncertainty.

Approach	What it does	Why it may not be suitable here
Pooled scalar \(t\) test	Tests the pooled estimate using total variance and pooled degrees of freedom.	It is the relevant approach for the scalar null described here, subject to its assumptions.
Standard normal test	Uses normal rather than \(t\) tail probabilities.	It can be a reasonable approximation with very large degrees of freedom, but can give misleading tail areas when degrees of freedom are small.
Separate tests in each imputation	Produces one test statistic or p-value per completed data set.	Those individual tests do not account for between-imputation uncertainty as a pooled scalar test does; their p-values should not simply be averaged.
Single completed-data analysis	Uses only one imputed data set for inference.	It discards the variation across imputations and does not represent the full imputation uncertainty.