Pooling Standardized Effects And Their Uncertainty

Why Standardized Effects Need Careful Pooling

A standardized contrast expresses a difference in outcome units relative to a standard deviation. For example, a mean difference of 5 points is easier to compare across outcomes when reported as a difference of 0.4 standard deviations. In multiply imputed data, however, both the contrast and the standard deviation used to scale it may be estimated, and their estimates can vary across imputations. Ignoring that uncertainty can understate the variance of the standardized effect.

The target must be defined before pooling. A standardized contrast can use a pooled within-group standard deviation, a baseline standard deviation, or another explicitly chosen reference scale. These choices need not estimate the same quantity. The standardization rule must be the same in every imputation; otherwise, the imputation-specific estimates do not represent a common estimand.

Definition: In imputation \(j\), let \(\Delta_j\) be an estimated contrast in outcome units and let \(S_j>0\) be the estimated standard deviation chosen to standardize it. The standardized effect is \(Q_j=\Delta_j/S_j\). Its estimated within-imputation variance \(U_j\) must account for sampling uncertainty in both \(\Delta_j\) and \(S_j\), including their covariance when it is nonzero.

The method below assumes the imputations are proper for the analysis, the same contrast and standardization rule are used in every imputation, and each \(U_j\) estimates the repeated-sampling variance of its corresponding \(Q_j\). It also assumes the standardized estimates are sufficiently well behaved for a normal-based pooling approximation. The imputation model must support valid inference for the analysis; the missing-data assumptions and imputation-model choices are addressed in the earlier material on missing data.

Within-Imputation Uncertainty in a Ratio

Because a standardized effect is a ratio, its variance is not generally just the variance of the contrast divided by a fixed constant. A first-order delta-method approximation makes the role of both estimated quantities explicit. Let the estimated vector be \((\Delta_j,S_j)\), with estimated covariance matrix having variances \(V_{\Delta,j}\), \(V_{S,j}\), and covariance \(C_j\). The gradient of \(g(\Delta,S)=\Delta/S\) is \((1/S,-\Delta/S^2)\). Applying the variance approximation gives:

U_j \approx \frac{V_{\Delta,j}}{S_j^2} +\frac{\Delta_j^2V_{S,j}}{S_j^4} -\frac{2\Delta_j C_j}{S_j^3}.

The second term represents uncertainty in the standardizing quantity. The final term is the covariance contribution; it can increase or decrease the variance depending on the sign of the covariance and contrast. If \(S_j\) is treated as fixed by the estimand or known without sampling uncertainty, then \(V_{S,j}=0\) and \(C_j=0\), leaving \(U_j=V_{\Delta,j}/S_j^2\). That simplification is not justified merely because software reports a standard deviation as a single number.

Justification: For small estimation errors, \(g(\widehat{\Delta},\widehat{S})\) is approximated by its first-order Taylor expansion around the underlying values. The variance of that linear approximation is the gradient multiplied by the covariance matrix and its transpose. Substituting the gradient \((1/S,-\Delta/S^2)\) yields the displayed formula.

When the standardizing quantity is itself derived from a variance estimate, its variance and covariance with the contrast can be obtained by another delta-method step or from the fitted model’s covariance estimates. The calculation should match the chosen definition of \(S_j\); formulas for a pooled within-group standard deviation, for example, are not automatically interchangeable with formulas for a baseline standard deviation.

Worked Example: Delta-Method Variance for One Imputation

Suppose, in a hypothetical imputed data set, the estimated contrast is \(\Delta_j=4.0\) outcome points and the chosen standard deviation is \(S_j=10.0\) points. Suppose the estimated variances are \(V_{\Delta,j}=4.00\) points squared and \(V_{S,j}=1.00\) points squared, and the estimated covariance is \(C_j=0.20\) points squared. The standardized estimate is:

Q_j=\frac{4.0}{10.0}=0.400.

Using the delta-method variance formula:

\begin{aligned} U_j &\approx \frac{4.00}{10.0^2} +\frac{4.0^2(1.00)}{10.0^4} -\frac{2(4.0)(0.20)}{10.0^3}\\ &=0.0400+0.0016-0.0016\\ &=0.0400. \end{aligned}

Thus the within-imputation standard error is \(\sqrt{0.0400}=0.200\) standardized-effect units. If the uncertainty in \(S_j\) were ignored, the variance would be \(4.00/10.0^2=0.0400\) in this particular example as well, because the positive scale-uncertainty term and negative covariance term cancel. Such cancellation is possible, but it should be computed rather than assumed.

Pool Estimates and Variances Across Imputations

For \(M\) imputations, calculate \(Q_j\) and \(U_j\) separately in each completed data set. The pooled point estimate is the average of the standardized estimates. The total variance combines the average within-imputation variance with the between-imputation variance, which measures how much the estimates differ across imputations:

\bar{Q}=\frac{1}{M}\sum_{j=1}^{M}Q_j,\qquad \bar{U}=\frac{1}{M}\sum_{j=1}^{M}U_j,\qquad B=\frac{1}{M-1}\sum_{j=1}^{M}(Q_j-\bar{Q})^2, \qquad T=\bar{U}+\left(1+\frac{1}{M}\right)B.

The pooled standard error is \(\sqrt{T}\). For a large complete-data degrees-of-freedom approximation, an interval uses a \(t\) critical value with:

\nu=(M-1)\left(1+\frac{\bar{U}}{(1+1/M)B}\right)^2, \qquad \bar{Q}\ \pm\ t_{\nu,\,1-\alpha/2}\sqrt{T}.

This degrees-of-freedom expression is an approximation; finite-sample settings may call for a small-sample adjustment. If \(B=0\), the displayed expression is not directly defined, so an appropriate complete-data or adjusted degrees-of-freedom approach is needed. The interval describes uncertainty in the common standardized estimand, under the analysis and imputation assumptions; it is not a probability statement about the fixed estimand.

Worked Example: Pooling Three Standardized Effects

Consider three hypothetical imputations. Each uses the same outcome contrast and the same standardization rule. The estimates and within-imputation variances, computed by the delta method, are as follows:

Imputation \(j\)	\(\Delta_j\) (points)	\(S_j\) (points)	\(Q_j=\Delta_j/S_j\)	\(U_j\)
1	3.0	10.0	0.300	0.0400
2	6.0	12.0	0.500	0.0400
3	3.6	9.0	0.400	0.0400

The pooled estimate is:

\bar{Q}=\frac{0.300+0.500+0.400}{3}=0.400.

The average within-imputation variance is \(\bar{U}=0.0400\). The between-imputation variance is:

B=\frac{(0.300-0.400)^2+(0.500-0.400)^2+(0.400-0.400)^2}{3-1} =\frac{0.0200}{2}=0.0100.

Therefore:

T=0.0400+\left(1+\frac{1}{3}\right)(0.0100)=0.05333, \qquad SE(\bar{Q})=\sqrt{0.05333}=0.2309.

The approximate degrees of freedom are:

\nu=2\left(1+\frac{0.0400}{(4/3)(0.0100)}\right)^2 =2(4.00)^2=32.0.

Using \(t_{32,0.975}\approx2.037\), the 95% interval is \(0.400\pm2.037(0.2309)\), or approximately \((-0.0703,0.870)\) standardized-effect units. Under the model and imputation assumptions, the data are compatible with effects ranging from a small negative standardized contrast to a positive contrast of about 0.87 standard deviations. This interval includes zero and is fairly wide, reflecting both within-imputation and between-imputation uncertainty.

Why the Order of Calculation Matters

In general, averaging standardized estimates is not the same calculation as dividing an averaged contrast by an averaged standard deviation. For imputation \(j\), the standardization is \(Q_j=\Delta_j/S_j\). Since division is nonlinear, changes in \(S_j\) affect each ratio before the ratios are averaged. The two calculations can differ, and they may target different quantities.

Worked Example: Averaging Ratios Is Not a Ratio of Averages

Suppose two hypothetical imputations give contrasts of 2 and 8 points and standard deviations of 4 and 16 points, respectively. Pooling the imputation-specific standardized estimates gives:

\bar{Q}=\frac{(2/4)+(8/16)}{2}=\frac{0.500+0.500}{2}=0.500.

The ratio of the averaged contrast to the averaged standard deviation is instead:

\frac{\bar{\Delta}}{\bar{S}} =\frac{(2+8)/2}{(4+16)/2} =\frac{5}{10}=0.500.

Here the results happen to match. Change the second standard deviation to 8 points while keeping its contrast at 8 points. The imputation-specific average is now \((0.500+1.000)/2=0.750\), whereas the ratio of averages is \(5/6=0.833\). The numerical difference arises because the two procedures weight the imputation-specific scales differently. Unless the estimand is explicitly defined as a ratio of pooled quantities, pooling the standardized effects requires calculating each ratio first and then applying Rubin’s rules to those estimates and their variances.

This comparison is not a claim that one ordering is universally correct. It is a reminder that an estimand must be specified. If the scientific target is a contrast standardized by a particular population scale, the analysis should estimate that target consistently in every imputation and account for uncertainty in the scale. Pooling a contrast and scale separately and taking their ratio may be appropriate for a separately defined target, but it does not automatically reproduce the pooled standardized effect.

Diagnostics, Alternatives, and Limitations

Before interpreting the pooled result, inspect the imputation-specific contrasts, standard deviations, standardized effects, and variances. Large variation in \(S_j\), very small values of \(S_j\), or a few extreme \(Q_j\) values can make the ratio unstable and the normal approximation poor. Confirm that the same reference scale and contrast are used in every imputation, and check that the within-imputation variance calculation includes any relevant covariance term.

A common alternative is to pool the unstandardized contrast and report it in its original units. This avoids uncertainty from estimated standardization but answers a different reporting question. Another approach is to estimate a single scale by design or use a scale defined independently of the analysis sample; if that scale is genuinely fixed and known for the inferential target, its sampling uncertainty is not part of \(U_j\). Simply replacing the within-imputation standard deviation by a fixed value for convenience changes the target or omits uncertainty.

The delta method is a local, first-order approximation. It can be unreliable when the standardizing quantity is poorly estimated, when its distribution is markedly skewed, or when the denominator can approach zero. In those circumstances, a normal-based interval may have poor coverage even if the pooling arithmetic is correct. A carefully specified alternative, such as a suitable transformation or a resampling analysis that repeats the relevant estimation steps, may be needed. Whichever approach is chosen, the same estimand must be preserved across imputations.

Common mistake: “The standard deviation is just a scaling constant, so I can divide the contrast’s variance by \(S_j^2\).” This is correct only when the standardizing quantity is fixed or its uncertainty is intentionally outside the inferential target. If \(S_j\) is estimated, its variance and its covariance with \(\Delta_j\) generally contribute to the variance of \(\Delta_j/S_j\).

The standardized estimate is also not a causal effect merely because it has been scaled: its interpretation depends on the study design, model, contrast, and assumptions. Report the standardization rule and the uncertainty method, not just a number labelled “standardized effect.”

Check Your Understanding

Answer each question, then compare with the brief solutions.

If \(S_j\) is estimated, which terms in the delta-method variance formula represent its uncertainty and its association with the contrast?
For \(M=4\), suppose \(Q_j\) values are \(0.2, 0.4, 0.4, 0.6\), and all \(U_j=0.04\). Calculate \(\bar{Q}\), \(B\), and \(T\).
When can the within-imputation variance reduce to \(V_{\Delta,j}/S_j^2\)?
Why is a ratio of pooled quantities not generally equivalent to pooling imputation-specific standardized effects?
Name two warning signs that a normal-based interval for a pooled standardized effect may be unreliable.

Solutions. (1) The term \(\Delta_j^2V_{S,j}/S_j^4\) represents uncertainty in the scale; \(-2\Delta_j C_j/S_j^3\) represents covariance with the contrast. (2) \(\bar{Q}=(0.2+0.4+0.4+0.6)/4=0.400\). The squared deviations sum to \(0.0800\), so \(B=0.0800/3=0.02667\). Then \(T=0.0400+(1+1/4)(0.02667)=0.07333\). (3) When \(S_j\) is fixed or known without sampling uncertainty, so \(V_{S,j}=0\) and \(C_j=0\). (4) Division is nonlinear, so averaging the ratios and dividing the averages generally give different values and can target different quantities. (5) Warning signs include highly variable or very small estimated standard deviations and extreme standardized estimates or strong skewness across imputations.