Large-Sample Degrees Of Freedom For Rubin’s Rules

Why Degrees of Freedom Matter After Pooling

Multiple imputation produces a pooled estimate and a pooled variance, but a standard error alone does not determine how uncertain an estimate is for inference. A reference distribution is also needed. Rubin’s conventional large-sample degrees of freedom provide that reference: they determine the tails of the t distribution used for scalar pooled inference when the complete-data degrees of freedom are treated as effectively infinite.

The key input is the relative increase in variance due to missingness. When between-imputation variation is large relative to the average within-imputation variance, there is more uncertainty attributable to missing information, and the reference t distribution has heavier tails. The degrees of freedom also depend on the number of imputations, so they reflect both missing-data uncertainty and the finite Monte Carlo set of imputations.

Variance Components and the Degrees-of-Freedom Formula

For a scalar estimand, let \(m\) be the number of imputations. In imputation \(i\), let \(Q_i\) be the estimate and \(U_i\) its estimated variance. The pooled estimate is \(\bar Q\), the average within-imputation variance is \(\bar U\), and the between-imputation variance is \(B\). Rubin’s total variance estimate is \(T\).

Definition: The relative increase in variance due to nonresponse is \(r=(1+1/m)B/\bar U\), provided \(\bar U>0\). The conventional large-sample Rubin degrees of freedom are \(\nu_{\text{old}}=(m-1)(1+1/r)^2\) when \(r>0\). If \(r=0\), the formula is interpreted by its limit as \(\nu_{\text{old}}=\infty\).

The quantities used here are calculated as follows. The finite-imputation factor \(1+1/m\) appears in both the total variance and the relative increase in variance.

\bar Q=\frac{1}{m}\sum_{i=1}^{m}Q_i,\qquad \bar U=\frac{1}{m}\sum_{i=1}^{m}U_i,\qquad B=\frac{1}{m-1}\sum_{i=1}^{m}(Q_i-\bar Q)^2

T=\bar U+\left(1+\frac{1}{m}\right)B,\qquad r=\frac{(1+1/m)B}{\bar U},\qquad \nu_{\text{old}}=(m-1)\left(1+\frac{1}{r}\right)^2

This is a large-sample degrees-of-freedom formula: it treats the complete-data degrees of freedom as infinite. It does not use a residual degrees of freedom from a particular regression or analysis. The next stage of the calculation uses \(\sqrt{T}\) as the pooled standard error. To test \(H_0: Q=Q_0\), the pooled statistic is

t_{\text{obs}}=\frac{\bar Q-Q_0}{\sqrt{T}}, \qquad t_{\text{obs}}\ \text{is compared with a }t\text{ distribution with }\nu_{\text{old}}\text{ degrees of freedom.}

For a two-sided test, the reference p-value is the probability, under that t distribution, of a statistic at least as far from zero as \(|t_{\text{obs}}|\). The degrees of freedom need not be an integer; the t distribution is defined for positive non-integer degrees of freedom as well.

When the Formula Is Appropriate

The formula is used for scalar inference under the usual large-sample multiple-imputation approximations. Its main conditions are that the imputations are proper for the inferential task, the analysis estimates the same scalar quantity in every completed data set, and the within-imputation variances are valid estimates of the corresponding complete-data variances. The complete-data estimator should also be approximately normal with a variance estimated adequately by \(U_i\). These conditions concern both the imputation and analysis; a degrees-of-freedom formula cannot repair a misspecified imputation model or an unsuitable analysis model.

The calculation also requires at least two imputations and a positive average within-imputation variance. With \(m=1\), \(B\) cannot be estimated. If \(\bar U=0\), the relative increase is undefined, so this formula cannot be applied by substituting zero. Very small \(\bar U\) can also make \(r\) and the resulting degrees of freedom unstable. Such cases require investigation rather than automatic use of the formula.

The conventional formula is not a finite-sample correction for limited complete-data information. If the complete-data analysis has few degrees of freedom, treating them as infinite can give an overly large reference degrees of freedom. The Barnard–Rubin small-sample adjustment addresses that separate issue; it should not be confused with the large-sample formula discussed here.

Worked Example: Moderate Relative Increase in Variance

Suppose five imputations give a pooled estimate \(\bar Q=10.0\), an average within-imputation variance \(\bar U=4.00\), and between-imputation variance \(B=2.00\). Assume the scalar estimates and variance estimates are comparable across imputations, the imputation procedure is appropriate, and the complete-data estimator is approximately normal. Calculate the relative increase, degrees of freedom, and pooled test statistic for \(H_0:Q=8\).

First calculate the finite-imputation factor, \(1+1/m=1+1/5=1.20\). The contribution of between-imputation variation to \(T\) is \(1.20(2.00)=2.40\), so

T=4.00+2.40=6.40,\qquad r=\frac{2.40}{4.00}=0.600

The degrees of freedom are

\nu_{\text{old}}=(5-1)\left(1+\frac{1}{0.600}\right)^2 =4(2.6667)^2 \approx 28.44

The pooled standard error is \(\sqrt{6.40}=2.530\). Thus the test statistic is

t_{\text{obs}}=\frac{10.0-8}{2.530}\approx 0.7906

Compare \(0.7906\) with a t distribution having about \(28.44\) degrees of freedom. The two-sided p-value is about \(0.436\), so this calculation does not provide evidence against \(H_0\) at the \(0.05\) significance level. The result is conditional on the stated model and approximation assumptions; the degrees of freedom do not establish that those assumptions are valid.

Worked Example: High Relative Increase in Variance

Consider a hypothetical scalar analysis with \(m=10\), \(\bar U=1.50\), \(B=6.00\), and \(\bar Q=1.20\). Suppose the estimand and analysis are consistent across imputations, the imputations are appropriate, and the complete-data estimator is approximately normal. Find \(r\), \(\nu_{\text{old}}\), and the test statistic for \(H_0:Q=0\).

The finite-imputation factor is \(1+1/10=1.10\). Therefore,

T=1.50+1.10(6.00)=8.10,\qquad r=\frac{1.10(6.00)}{1.50}=\frac{6.60}{1.50}=4.40

Then

\nu_{\text{old}}=(10-1)\left(1+\frac{1}{4.40}\right)^2 =9(1.2273)^2 \approx 13.56

The pooled standard error is \(\sqrt{8.10}=2.846\), giving

t_{\text{obs}}=\frac{1.20-0}{2.846}\approx 0.4217

A two-sided comparison with \(t_{13.56}\) gives a p-value of about \(0.680\). Although there are ten imputations, the relative increase is high: the between-imputation contribution to total variance is large compared with \(\bar U\). The degrees of freedom are consequently much lower than they would be with little between-imputation variation.

Why the Degrees of Freedom Change

The formula’s behavior can be checked directly by rewriting it when \(r>0\):

\nu_{\text{old}}=(m-1)\frac{(1+r)^2}{r^2}

Holding \(m\) fixed, as \(r\) approaches zero, \((1+r)^2/r^2\) grows without bound, so \(\nu_{\text{old}}\) approaches infinity. In this limit, the t reference distribution approaches the standard normal distribution. As \(r\) becomes very large, \((1+r)^2/r^2=(1+1/r)^2\) approaches 1, so \(\nu_{\text{old}}\) approaches \(m-1\). This lower limiting value reflects the fact that estimating between-imputation variability relies on the \(m\) imputed estimates.

These limits explain the direction of the adjustment: more relative between-imputation uncertainty leads to fewer degrees of freedom and heavier tails. Increasing \(m\) can increase the degrees of freedom for a fixed \(r\), but it does not automatically make \(r\) small. More imputations reduce Monte Carlo error in the pooling calculations; they do not remove the uncertainty arising from missing data.

A short algebraic check also shows why using the wrong variance ratio matters. The variance contribution from imputation is \((1+1/m)B\), not \(B\) alone. Therefore \(r\) must be calculated using that same corrected contribution. Omitting \(1+1/m\) changes both \(r\) and \(\nu_{\text{old}}\), even when the pooled point estimate is unchanged.

Common Mistakes, Diagnostics, and Alternatives

Using \(B/\bar U\) as \(r\). This leaves out the finite-imputation factor. Use \(r=(1+1/m)B/\bar U\), consistent with the definition of \(T\).
Using a normal reference automatically. A normal reference ignores the extra uncertainty represented by finite degrees of freedom. Use the t reference with the calculated \(\nu_{\text{old}}\) under the large-sample assumptions.
Setting the degrees of freedom to \(m-1\) in every analysis. The quantity \(m-1\) is not the conventional formula in general; it is approached as \(r\) becomes very large. Calculate the formula using the observed \(r\).
Assuming many imputations guarantee large degrees of freedom. The formula depends on both \(m\) and \(r\). Check the variance components rather than judging the reference distribution from \(m\) alone.
Treating the result as a diagnostic of the imputation model. A numerical degrees of freedom can be calculated even if the model is inadequate. Inspect the imputation and analysis for compatibility, convergence problems, implausible values, or unusually variable estimates.

The main alternatives answer different questions. A normal reference is a limiting approximation when degrees of freedom are very large, not a substitute that captures finite-imputation uncertainty. A fixed \(m-1\) reference ignores how large or small the relative increase in variance is. For settings with limited complete-data degrees of freedom, the Barnard–Rubin adjustment is a small-sample alternative; it incorporates information that the conventional large-sample formula deliberately treats as infinite.

The conventional formula is useful because it is straightforward and links pooled t inference directly to the relative increase in variance. Its limitation is equally important: its validity rests on large-sample approximations and a suitable imputation-and-analysis procedure. A calculated value such as \(28.44\) is not a measure of the number of observed individuals, nor does it replace careful assessment of model assumptions.

Worked Example: Zero Estimated Between-Imputation Variance

Suppose four hypothetical imputations produce \(\bar Q=1.50\), \(\bar U=0.250\), and \(B=0\). Assume the same scalar estimand is analyzed in each imputation, the within-imputation variance is positive and valid, and the large-sample normal approximation is reasonable. Find the limiting degrees of freedom and the pooled test statistic for \(H_0:Q=0\).

Because \(B=0\), the between-imputation contribution to total variance is \((1+1/4)0=0\). Thus \(T=\bar U=0.250\), and \(r=0/0.250=0\). The degrees-of-freedom formula is interpreted by its limit as \(r\) approaches zero from above, giving \(\nu_{\text{old}}=\infty\). The pooled standard error and test statistic are

\sqrt{T}=\sqrt{0.250}=0.500,\qquad t_{\text{obs}}=\frac{1.50-0}{0.500}=3.00

With infinite degrees of freedom, the reference distribution is the standard normal limit; the two-sided p-value is approximately \(0.00270\). In practice, an estimated \(B=0\) from a small number of imputations may reflect limited Monte Carlo resolution rather than genuine absence of between-imputation uncertainty. It is a reason to inspect the imputations and, if appropriate, increase their number—not a general reason to assume missing-data uncertainty is absent.

Check Your Understanding

Use the conventional large-sample formula. Treat any stated large-sample assumptions as satisfied, and give the arithmetic before interpreting the result.

With \(m=5\), \(\bar U=2.00\), and \(B=1.00\), calculate \(r\) and \(\nu_{\text{old}}\).
With \(m=8\), \(\bar U=3.00\), and \(B=0\), what is the limiting degrees of freedom?
For \(m=4\), \(\bar U=1.00\), and \(B=2.00\), calculate \(T\), \(r\), and \(\nu_{\text{old}}\).
Explain why increasing the number of imputations does not, by itself, eliminate missing-data uncertainty.

Solutions. (1) \(r=(1+1/5)(1.00)/2.00=0.600\), so \(\nu_{\text{old}}=4(1+1/0.600)^2=28.44\). (2) Here \(r=0\); by the limiting convention, \(\nu_{\text{old}}=\infty\). (3) \(T=1.00+(1+1/4)(2.00)=3.50\), \(r=2.50/1.00=2.50\), and \(\nu_{\text{old}}=3(1+1/2.50)^2=5.880\). (4) More imputations can reduce Monte Carlo error in the pooling calculations, but they do not make the missing values known or erase the between-imputation uncertainty. The degrees of freedom depend on both \(m\) and the relative increase in variance.