Interpreting Confidence Intervals Across Delta Values

Why Compare the Intervals, Not Just the Estimates?

A sensitivity curve can show how a pooled estimate moves as a delta assumption changes. But the estimate is only part of the inferential result. A delta-adjusted scenario can also change the variability within completed data sets, the variation between their estimates, or both. Consequently, two scenarios may have different pooled estimates but similar interval widths, or similar estimates but substantially different widths.

A confidence interval should therefore be read along two dimensions: its location, which reflects the pooled estimate, and its width, which reflects estimated uncertainty under that scenario. Comparing these dimensions helps explain how the inferential conclusion responds to the missing-data assumptions. It does not establish which assumption is correct: delta values encode assumptions about unobserved outcomes, and the observed data alone generally cannot determine them.

Definition: For a specified delta scenario, let \(Q_j\) be the estimate from imputed data set \(j\), and let \(U_j\) be its estimated variance. With \(m\) imputations, the pooled estimate is \(\bar{Q}=m^{-1}\sum_{j=1}^{m}Q_j\), the average within-imputation variance is \(\bar{U}=m^{-1}\sum_{j=1}^{m}U_j\), and the between-imputation variance is \(B=(m-1)^{-1}\sum_{j=1}^{m}(Q_j-\bar{Q})^2\). Rubin’s total variance is \(T=\bar{U}+(1+1/m)B\), and the pooled standard error is \(SE=\sqrt{T}\). A two-sided \(100(1-\alpha)\%\) pooled confidence interval is \(\bar{Q}\mathbin{\pm}t_{\nu,1-\alpha/2}SE\), using an appropriate degrees-of-freedom estimate \(\nu\).

The interval is calculated separately within each delta scenario. In the usual large-complete-data-degrees-of-freedom approximation, define the relative increase in variance due to nonresponse as \(r=(1+1/m)B/\bar{U}\). Rubin’s degrees-of-freedom approximation is then \(\nu=(m-1)(1+1/r)^2\). When the complete-data degrees of freedom are not effectively large, a finite-sample adjustment such as the Barnard–Rubin adjustment may be needed. The approximation, variance calculation, and critical value should be applied consistently across the scenarios being compared.

This calculation presumes that each scenario’s imputation and analysis are suitable for the estimand, that the imputations adequately represent uncertainty under the specified imputation model, and that the analysis within each completed data set estimates the same target quantity. Delta adjustment adds an explicit assumption about the distribution of missing values relative to the reference imputation model. It does not make that assumption empirically identified. The interval describes sampling and imputation uncertainty conditional on the scenario; it does not account for uncertainty about which delta scenario is appropriate.

Read Location and Width Separately

The pooled estimate is the center of the interval. If it changes across delta values, the center’s movement is a change in the estimated effect under the altered missing-data assumption. The interval width is \(2t_{\nu,1-\alpha/2}SE\). It can change because \(SE\) changes, because the degrees of freedom change the critical value, or both. Since \(SE\) depends on both \(\bar{U}\) and \(B\), a width change is not automatically evidence that the missing values have become more variable in a simple substantive sense.

A useful comparison is to report, for every delta scenario, the pooled estimate, pooled standard error, degrees of freedom, interval endpoints, and interval width. Calculate width from the unrounded endpoints or directly from the formula, not by subtracting endpoints already rounded for display. If the scientific null value is zero for a difference, whether the interval includes zero describes the corresponding two-sided test at the matching level under the model and approximation. It is not a measure of the probability that the null hypothesis is true.

Worked Example: Estimate Shift and Width Change Together

Suppose a hypothetical analysis estimates an adjusted mean difference in a symptom score, in points, under two delta scenarios. There are \(m=20\) imputations in each scenario, and both intervals are two-sided 95% intervals. In scenario A, \(\bar{Q}=2.400\), \(\bar{U}=0.5163\) points squared, and \(B=0.2017\) points squared. In scenario B, \(\bar{Q}=1.800\), \(\bar{U}=0.6400\) points squared, and \(B=0.2500\) points squared. Assume the large-complete-data-degrees-of-freedom approximation is appropriate.

For scenario A, \(T=0.5163+(1+1/20)(0.2017)=0.5163+1.05(0.2017)=0.728085\) points squared, so \(SE=\sqrt{0.728085}=0.8533\) points. Its relative increase is \(r=1.05(0.2017)/0.5163=0.4102\), giving \(\nu=19(1+1/0.4102)^2\approx224.6\). Using \(t_{224.6,0.975}\approx1.9706\), the interval is \(2.400\mathbin{\pm}1.9706(0.8533)\), or approximately \(2.400\mathbin{\pm}1.6815\) points. The endpoints are about \(0.7185\) and \(4.0815\), and the width calculated before rounding is about \(3.363\) points.

For scenario B, \(T=0.6400+1.05(0.2500)=0.9025\) points squared, and \(SE=\sqrt{0.9025}=0.9500\) points. Here \(r=1.05(0.2500)/0.6400=0.4102\), so \(\nu\approx224.6\) again. The interval is \(1.800\mathbin{\pm}1.9706(0.9500)\), or approximately \(1.800\mathbin{\pm}1.8721\) points. Its endpoints are about \(-0.0721\) and \(3.6721\), and its width is about \(3.744\) points.

The pooled estimate has decreased by \(0.600\) points, while the interval has widened from about \(3.363\) to \(3.744\) points. The center shift reflects a changed estimated difference under the alternative delta assumption; the width change reflects changed pooled uncertainty. The scenario B interval includes zero, whereas scenario A’s does not. This is a statement about the scenario-specific 95% intervals, not proof that one delta value is correct or that the effect has changed in the population.

Justifying the Width Comparison

The interval formula makes clear why location and uncertainty need separate summaries. Let \(c=t_{\nu,1-\alpha/2}\), which is positive for the usual two-sided confidence levels. The lower and upper endpoints are \(\bar{Q}-cSE\) and \(\bar{Q}+cSE\). Subtracting the lower endpoint from the upper gives:

(\bar{Q}+cSE)-(\bar{Q}-cSE)=2cSE.

The center, \(\bar{Q}\), cancels. Thus an estimate shift alone cannot change the width if \(SE\) and \(\nu\) stay fixed. Conversely, changing \(SE\) or \(\nu\) can change width even when the pooled estimate stays fixed. This is why reporting only endpoints, or only the pooled estimate, can obscure what changed across delta scenarios.

Worked Example: A Shifted Estimate with Unchanged Uncertainty

Suppose two hypothetical delta scenarios estimate an adjusted mean difference in a scale score, in points. In both, \(SE=0.400\) points, \(\nu=40\), and the two-sided 95% critical value is approximately \(2.021\). The pooled estimate is \(1.200\) points in scenario C and \(0.900\) points in scenario D.

In scenario C, the interval is \(1.200\mathbin{\pm}2.021(0.400)=1.200\mathbin{\pm}0.8084\), giving approximately \((0.3916,2.0084)\) points. In scenario D, it is \(0.900\mathbin{\pm}0.8084\), giving approximately \((0.0916,1.7084)\) points. Each interval has width \(2(0.8084)=1.617\) points.

Here the location changes by \(0.300\) points but the estimated uncertainty, as reflected in interval width, does not. Both intervals lie above zero. That shared feature does not show that the estimates are identical; it says that each scenario’s interval excludes zero under its own assumptions and calculation.

Worked Example: Same Estimate with Different Uncertainty

Consider two more hypothetical scenarios for an adjusted mean difference in a blood-pressure measure, in mm Hg. Both produce \(\bar{Q}=0.350\) mm Hg and \(\nu=50\), for which \(t_{50,0.975}\approx2.009\). Scenario E has \(SE=0.200\) mm Hg; scenario F has \(SE=0.300\) mm Hg.

For scenario E, the half-width is \(2.009(0.200)=0.4018\) mm Hg, so the interval is \(0.350\mathbin{\pm}0.4018=(-0.0518,0.7518)\) mm Hg, with width \(0.8036\) mm Hg. For scenario F, the half-width is \(2.009(0.300)=0.6027\) mm Hg, so the interval is \(0.350\mathbin{\pm}0.6027=(-0.2527,0.9527)\) mm Hg, with width \(1.2054\) mm Hg.

The point estimate is unchanged, but the interval is wider in scenario F because its standard error is larger. In a MICE analysis, this difference can arise through the within-imputation variance, the between-imputation variance, or both. The interval comparison alone does not identify which component changed; inspect \(\bar{U}\), \(B\), and the degrees of freedom alongside the endpoints.

Diagnostics, Comparisons, and Limits

For each delta value, check that the same estimand, analysis model, confidence level, and pooling method are being used. Then inspect the components of the pooled variance. A sharp change in \(B\) may indicate that the estimates vary more across imputations in that scenario; a change in \(\bar{U}\) may reflect differences in complete-data uncertainty. A change in \(\nu\) can affect the critical value as well. The earlier material on implied distributions and boundary effects is also relevant: an interval can be calculated correctly while the delta scenario generates implausible imputed values or pushes predictions toward a boundary.

Check that the imputation chains have mixed adequately and that \(m\) is sufficient for stable estimates of between-imputation uncertainty. Monte Carlo variability can make adjacent scenarios appear to have slightly different pooled estimates or widths even when the underlying sensitivity pattern is smooth. When the differences are small, examine simulation error and consider increasing the number of imputations before interpreting fine detail. Also verify that the imputation and analysis procedures are appropriate for the outcome and estimand; a narrow interval does not repair model misspecification.

A sensitivity curve of pooled estimates emphasizes how the center changes with delta. Confidence intervals add scenario-specific uncertainty, but comparing whether two intervals overlap is not a formal test of whether their pooled estimates differ. The intervals are computed under alternative assumptions applied to the same underlying study, not as independent-study intervals. Nor does a set of intervals average over uncertainty about delta. A normal-based interval may be a reasonable approximation with large degrees of freedom, but when the imputation degrees of freedom are limited, the \(t\) critical value matters; use the degrees-of-freedom method appropriate to the analysis.

These comparisons are useful for describing robustness: do plausible assumptions lead to similar effect estimates, similar precision, and similar substantive interpretation? They do not determine which scenario is most credible. In particular, observing that an interval includes or excludes a null value at selected deltas is not a substitute for justifying the range of deltas. Locating the value at which statistical significance changes is a distinct analysis, not a conclusion to infer from interval widths alone.

Common Mistakes

Calling every wider interval a shifted effect. A wider interval can result from a larger standard error or a changed degrees-of-freedom critical value, even if the pooled estimate is unchanged. Report center and width separately.
Calling an estimate shift an uncertainty change. If the interval moves without changing width, the center changed while the estimated uncertainty did not, as in the second worked example.
Comparing rounded endpoints to obtain a width. Rounding each endpoint first can produce a width that differs in the last reported digit. Calculate width from unrounded quantities, then round the result.
Treating overlap as a test of scenario differences. Overlapping scenario-specific intervals do not establish that the estimates are equal, and non-overlap is not the general test for a difference. These intervals answer questions within scenarios, not a paired comparison of scenarios.
Interpreting an interval as uncertainty about delta. The interval is conditional on the stated delta scenario. It does not incorporate uncertainty about which delta value is plausible.

Key takeaway: For each delta scenario, report the pooled estimate and its interval, then compare interval centers and widths separately. Width depends on pooled uncertainty and the critical value; neither a changed interval nor a change in whether it includes the null identifies the correct missing-data assumption.

Check Your Understanding

Answer each question using the scenario-specific interpretation of a pooled interval.

If the pooled estimate changes across delta values but \(SE\) and degrees of freedom remain fixed, what happens to interval width?
A 95% interval for a mean difference is \((-0.4,1.2)\) points. State what its inclusion of zero means, and one thing it does not mean.
Why should interval width be calculated before rounding the endpoints?
Two delta scenarios have the same pooled estimate, but one has a larger \(SE\) and the same degrees of freedom. Which interval is wider, and why?

Solutions. (1) Width is unchanged because it equals \(2t_{\nu,0.975}SE\); only the center moves. (2) Under the stated scenario and interval method, the 95% interval includes zero, so it does not exclude a zero mean difference at the corresponding two-sided 5% level. It does not mean there is a 95% probability that the true difference is in this particular interval. (3) Rounded endpoints can alter their displayed difference, so calculate from unrounded endpoints or directly from the interval formula. (4) The scenario with larger \(SE\) has the wider interval because the critical value is unchanged and width is \(2t_{\nu,0.975}SE\).