Comparing Observed and Imputed Binary Proportions

Why Compare Binary Proportions?

For a binary variable, an observed-imputed comparison asks whether the event occurs at similar rates among cells with recorded values and cells whose values were filled in by MICE. For example, a health dataset might record whether a hypothetical participant experienced an event, coded as 1 for yes and 0 for no. Comparing the proportions coded 1 can reveal a striking discrepancy that is harder to see by inspecting a table of imputations alone.

The comparison is a diagnostic, not a requirement that the two proportions match. Under an appropriate imputation model, imputed values can reflect relationships with other variables. If the people with missing values differ in their predictor values from those with observed values, their marginal event proportions may reasonably differ. The aim is to describe and investigate the difference, not to treat equality as a universal benchmark.

Definition: Let \(Y\) be a binary variable coded 1 for the event and 0 otherwise. Let \(n_{\mathrm{obs}}\) be the number of observed cells, \(m\) the number of missing cells, and \(n=n_{\mathrm{obs}}+m\). The observed event proportion is \(\hat p_{\mathrm{obs}}=e_{\mathrm{obs}}/n_{\mathrm{obs}}\), where \(e_{\mathrm{obs}}\) is the observed number of events. In completed dataset \(\ell\), the imputed event proportion is \(\hat p_{\mathrm{imp},\ell}=e_{\mathrm{imp},\ell}/m\), where \(e_{\mathrm{imp},\ell}\) is the number of imputed events among those \(m\) cells. The signed gap is \(\Delta_\ell=\hat p_{\mathrm{imp},\ell}-\hat p_{\mathrm{obs}}\). The missingness fraction is \(r=m/n\).

A positive gap means the imputed cells have a higher event proportion; a negative gap means they have a lower one. When there are \(M\) completed datasets, report the \(M\) imputed proportions or summarize them by their mean, range, and, when useful, standard deviation. Each completed dataset supplies one realization of the missing values. The \(M\) proportions are not \(M\) independent samples of observed people.

Relating the Gap to the Completed-Data Proportion

The size of a gap alone does not show how much the imputed values change the event proportion across all \(n\) records. The missingness fraction provides that context. In a completed dataset, the total number of events is the observed event count plus the imputed event count. Dividing by the full sample size gives:

\hat p_{\mathrm{complete},\ell} =\frac{e_{\mathrm{obs}}+e_{\mathrm{imp},\ell}}{n} =\frac{n_{\mathrm{obs}}\hat p_{\mathrm{obs}}+m\hat p_{\mathrm{imp},\ell}}{n}.

Since \(n_{\mathrm{obs}}/n=1-r\) and \(m/n=r\), subtracting the observed proportion yields:

\hat p_{\mathrm{complete},\ell}-\hat p_{\mathrm{obs}} =(1-r)\hat p_{\mathrm{obs}}+r\hat p_{\mathrm{imp},\ell}-\hat p_{\mathrm{obs}} =r(\hat p_{\mathrm{imp},\ell}-\hat p_{\mathrm{obs}}) =r\Delta_\ell.

This identity is a useful scale check. A gap of 0.10 shifts the completed-data proportion by 0.02 if 20% of cells are missing, but by 0.06 if 60% are missing. It does not say whether the imputation model is correct; it quantifies the arithmetic contribution of the imputed cells to this particular overall proportion.

Worked Example: A Gap With Moderate Missingness

In a hypothetical dataset, a binary outcome is observed for 120 of 150 people. There are 36 observed events. In one completed dataset, MICE imputes 12 events among the 30 missing cells. Calculate both proportions, their gap, and the completed-data proportion.

The observed event proportion is \(36/120=0.300\). The imputed event proportion is \(12/30=0.400\). Thus the signed gap is \(0.400-0.300=0.100\), or 10 percentage points. The missingness fraction is \(30/150=0.200\).

The completed-data event count is \(36+12=48\), so its event proportion is \(48/150=0.320\). The identity gives the same result as a shift from the observed proportion: \(r\Delta=0.200(0.100)=0.020\), and \(0.300+0.020=0.320\). Thus the completed-data proportion is 2 percentage points above the observed-cell proportion.

This calculation describes one completed dataset. With multiple imputations, the imputed count may vary between datasets, so the completed-data proportion and gap should also be examined across imputations.

Observed Sample Size and Uncertainty

The observed proportion is estimated from \(n_{\mathrm{obs}}\) values, not from all \(n\) records. Under the simple binomial sampling model in which observed values are independent draws with a common event probability \(p\), the standard error of \(\hat p_{\mathrm{obs}}\) is estimated by:

SE(\hat p_{\mathrm{obs}}) =\sqrt{\frac{\hat p_{\mathrm{obs}}(1-\hat p_{\mathrm{obs}})}{n_{\mathrm{obs}}}}.

This formula describes sampling variability under that binomial model. It does not account for selection into the observed cells, uncertainty about the imputation model, or variation among imputed datasets. In particular, it is not the standard error of the observed-imputed gap. Treating the imputed cells as a second independent sample and using a two-sample test would generally ignore how those values were generated and the predictors used in imputation.

Worked Example: A Large Gap With Many Missing Cells

Suppose a hypothetical binary variable has 8 events among 80 observed cells and 120 missing cells. In one completed dataset, 42 of the missing cells are imputed as events. Calculate the proportions and use the observed sample size to describe the observed proportion's sampling variability.

The observed proportion is \(8/80=0.100\); the imputed proportion is \(42/120=0.350\). The gap is \(0.350-0.100=0.250\), or 25 percentage points. The missingness fraction is \(120/200=0.600\). The completed-data proportion is \((8+42)/200=0.250\), a shift of 0.150 from the observed proportion. This agrees with \(r\Delta=0.600(0.250)=0.150\).

Under the independent binomial model for the observed cells, the estimated standard error is:

SE(\hat p_{\mathrm{obs}}) =\sqrt{\frac{0.100(1-0.100)}{80}} =\sqrt{0.001125} \approx 0.0335.

The observed proportion therefore has an estimated standard error of about 3.35 percentage points under that model. The 25-point observed-imputed gap is large relative to this scale, so it warrants investigation. However, dividing the gap by this standard error is not a valid significance test of imputation quality: the gap is not a difference between two independent binomial samples, and the observed and missing groups may differ in their predictor composition.

Reading the Comparison Across Imputations

A single completed dataset can give a misleadingly sharp impression if the imputed event count varies across imputations. Summarize the imputed proportions across the \(M\) completed datasets and compare their variation with the observed proportion. The following hypothetical example shows why both the average gap and between-imputation variation matter.

Worked Example: Summarizing Several Imputations

A hypothetical dataset has 100 observed cells, of which 25 are events, and 50 missing cells. Across four completed datasets, the imputed event proportions are 0.27, 0.30, 0.33, and 0.30. Find the mean imputed proportion, the mean gap, the missingness fraction, and the mean completed-data proportion.

The observed proportion is \(25/100=0.250\). The mean imputed proportion is \((0.27+0.30+0.33+0.30)/4=1.20/4=0.300\). The mean gap is \(0.300-0.250=0.050\), or 5 percentage points. The imputed proportions range from 0.27 to 0.33, a range of 0.06.

The missingness fraction is \(50/150=0.333\) to three significant figures. The average completed-data proportion is \((25+50(0.300))/150=40/150=0.267\). The identity gives the same shift: \(0.333(0.050)\approx0.0167\), and \(0.250+0.0167\approx0.267\). The mean gap is modest, and the imputed proportion varies by 6 percentage points across these four datasets. Four values are too few to characterize that variation precisely, but the calculation illustrates why the imputation-specific proportions should not be hidden behind one completed dataset.

Assumptions, Diagnostics, and Alternatives

The comparison is meaningful when the binary variable has a consistent definition and coding in observed and imputed cells, and the denominators are correctly identified. The imputation model should be appropriate for the variable and include relevant information needed for the analysis. MICE chains should also have reached a stable distribution before their imputations are summarized; the chain diagnostics covered earlier help assess that stability. A stable chain does not by itself establish that the imputation model is well specified.

Do not assume that the observed and imputed cells must have the same marginal event proportion. Under a plausible missing-at-random analysis, observed and missing cells may have different covariate distributions, and their event proportions can differ because the imputation model uses those covariates. A marginal comparison is therefore a useful warning signal, not a direct test of the missingness assumption or a proof of model failure. Conversely, similar proportions do not prove that the imputation model is adequate.

When interpreting a discrepancy, check the coding, denominators, number of observed events, missingness fraction, and variation among imputations. Then consider whether the observed and missing cells differ in ways represented by variables in the imputation model. A very small observed sample makes the observed proportion imprecise; a large missingness fraction makes a given gap more influential on the overall completed-data proportion. If the observed proportion is exactly 0 or 1, the plug-in standard error above is zero, which should not be interpreted as certainty: it can instead reflect a small or sparse observed sample.

This diagnostic complements, rather than replaces, trace plots of imputed binary proportions and other checks of imputation behavior. A formal comparison that treats the observed and imputed proportions as independent samples is usually inappropriate. The imputed values are generated conditionally by a model, and the two groups need not be exchangeable. Comparisons of nominal categories and subgroup-specific comparisons answer related but distinct questions; the overall binary comparison here should not be expanded into a general test of categorical distributions.

Key takeaway: Report the observed and imputed event proportions, their signed gap, the missingness fraction, and variation across imputations. Use \(r\Delta\) to show the gap's arithmetic impact on the overall completed-data proportion, while treating the comparison as a diagnostic rather than a pass-or-fail test.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

There are 60 observed cells, with 18 events, and 40 missing cells, with 16 imputed events in one completed dataset. Find the observed and imputed proportions, signed gap, missingness fraction, and completed-data proportion.
If the signed gap is 0.08 and the missingness fraction is 0.25, what is the shift in the completed-data proportion relative to the observed proportion?
Why is it generally inappropriate to use a two-sample test that treats observed and imputed cells as independent samples?
What does a large observed-imputed gap establish, and what does it not establish?

Solutions. (1) The observed proportion is \(18/60=0.300\); the imputed proportion is \(16/40=0.400\); the gap is \(0.400-0.300=0.100\). The missingness fraction is \(40/100=0.400\), and the completed-data proportion is \((18+16)/100=0.340\), which is \(0.300+0.400(0.100)\). (2) The shift is \(r\Delta=0.25(0.08)=0.020\), or 2 percentage points. (3) Imputed values are model-generated conditional draws, not an independent sample of observed people; the groups may also differ in predictor composition. (4) It establishes a descriptive difference in the two marginal proportions for the chosen data and imputations. It does not, by itself, establish that the imputation model is invalid or that the missingness assumption fails.