Conditional Imputation Within Meaningful Groups

Why Impute Within Groups?

A single overall summary can be a poor substitute for a missing value when the variable differs systematically across meaningful groups. For example, a missing service cost for a particular repair category may be better represented by costs for that category than by the average across all categories. Group-specific imputation uses information about an observed grouping variable to make the replacement more conditional on the case.

The group must have a defensible role: it should be known for the case being imputed and related to the variable being filled for a reason that makes sense in the data. A group label should not be selected merely because it happens to produce convenient values. Even a well-chosen group does not guarantee that imputation is unbiased. The observed cases within a group must be informative about the missing cases in that group.

Definition: Let $Y$ be a variable with missing values and $G$ a categorical grouping variable. Group-specific imputation replaces a missing $Y$ for a case in group $g$ with a summary calculated from observed $Y$ values in that group. For a group-specific mean, the replacement is $\bar{Y}_{g,\mathrm{obs}}$, the mean of observed values in group $g$; for a group-specific median, it is $\tilde{Y}_{g,\mathrm{obs}}$, the median of those values.

This is a form of single imputation: each missing value is filled with one estimated value. A basic working assumption is that, within each group, observed values represent the distribution of values that would have been observed for cases with missing $Y$. In conditional-independence notation, this can be expressed as $Y \perp R \mid G$, where $R$ indicates whether $Y$ is observed. This condition is not established just by grouping the data. If missingness still depends on the unobserved value after conditioning on $G$, group summaries can remain systematically wrong.

Group means and medians also serve different purposes. The mean is natural when squared deviations matter and values are not dominated by extreme observations. The median is a robust center and can be suitable when absolute deviations matter or the group distribution is skewed. Neither summary recreates the full variation in the missing values: every missing case in a group receives the same value under this simple method.

Choosing the Summary and Fallback Rule

Before calculating replacements, specify the variable, the group, the summary, and a rule for groups that do not have enough observed values. A group with one observed value has a calculable mean and median, but that statistic may be an unreliable estimate of the group center. A minimum group size is a practical safeguard, not a universal statistical constant; its value should be chosen and documented for the application.

A fallback is a prespecified replacement used when a group summary is unavailable or fails the minimum-size rule. For example, a global median calculated from observed values might be used when a group has too few observations. The fallback should be calculated from the relevant analysis data only. If no overall observed values are available either, there is no data-based fallback; the analyst must resolve the problem rather than silently invent a replacement.

Check the grouping variable.
Confirm that the group is meaningful, observed at prediction time if needed, and not simply a proxy that would be unavailable for future cases.

Calculate observed group summaries.
For every group, record the number of observed values and calculate the chosen summary from those values only.

Apply the eligibility rule.
Use a group summary only if the group has enough observed values. Otherwise, apply the stated fallback.

Preserve the audit trail.
Keep the original variable, record which values were missing, and document the summary, minimum group size, and fallback.

Worked Calculations

Worked Example: Group Means for Repair Costs

A hypothetical maintenance database records repair cost in dollars and repair category. In category A, the observed costs are 120, 150, and 180; a fourth cost is missing. In category B, the observed costs are 60 and 90; a third cost is missing. Use the observed category means to impute the missing costs, then compare them with the overall observed mean.

Method. Use each category's observed mean. No minimum group-size fallback is specified, so use the calculated means. The observed values are the only values used in the summaries.

Category A. Its observed total is $120+150+180=450$, and it has 3 observed costs. Its mean is $450/3=150$. Impute 150 dollars for the missing category A cost.

Category B. Its observed total is $60+90=150$, and it has 2 observed costs. Its mean is $150/2=75$. Impute 75 dollars for the missing category B cost.

Overall comparison. There are 5 observed costs, with total $450+150=600$. The overall observed mean is $600/5=120$ dollars. An overall-mean imputation would assign 120 dollars to both missing costs, whereas the group-specific replacements are 150 dollars and 75 dollars.

Check. The category A replacement is between 120 and 180, and the category B replacement is between 60 and 90, as expected for means of these observed values. The method avoids the mistake of using one pooled average despite different observed cost levels between categories.

Worked Example: A Small Group and a Global-Median Fallback

A hypothetical portfolio has recorded claim amounts in three risk bands. Band L has observed amounts of 5 and 7 and one missing amount. Band M has observed amounts of 10, 12, and 14 and one missing amount. Band H has an observed amount of 30 and one missing amount. Use group medians only when the group has at least 3 observed amounts; otherwise use the global median of all observed amounts.

Step 1: Calculate the fallback. The six observed amounts, sorted, are 5, 7, 10, 12, 14, 30. With an even number of observations, the median is the average of the two middle values: $(10+12)/2=11$. Thus the fallback is 11.

Step 2: Evaluate band L. Band L has $n_L=2$, which is below the required minimum of 3. Its own median is $(5+7)/2=6$, but the rule says not to use that estimate. Impute the fallback, 11.

Step 3: Evaluate band M. Band M has $n_M=3$, so its group summary is eligible. Its sorted values are 10, 12, and 14, with median 12. Impute 12.

Step 4: Evaluate band H. Band H has $n_H=1$, so its group summary is not eligible. Impute the fallback, 11, rather than using its single observed amount of 30.

Check. The completed missing values are 11 for band L, 12 for band M, and 11 for band H. A common mistake is to use the median of every group regardless of size, which would put 6 in band L and 30 in band H. Those values violate the stated minimum-size rule; the fallback must be applied consistently.

Why the Mean Matches a Squared-Error Objective

The mean is not an arbitrary group summary when the objective is to minimize expected squared error. Suppose $Y$ has a finite conditional second moment in group $g$. Consider predicting $Y$ for a case in that group by a constant $c$. The conditional expected squared error is minimized when $c$ is the conditional mean. To see why, write $\mu_g=E[Y\mid G=g]$ and expand:

E[(Y-c)^2\mid G=g] =E[(Y-\mu_g+\mu_g-c)^2\mid G=g]

=E[(Y-\mu_g)^2\mid G=g]+(\mu_g-c)^2 +2(\mu_g-c)E[Y-\mu_g\mid G=g].

By the definition of $\mu_g$, $E[Y-\mu_g\mid G=g]=0$. Therefore,

E[(Y-c)^2\mid G=g] =\operatorname{Var}(Y\mid G=g)+(\mu_g-c)^2.

The variance term does not depend on $c$, while the squared difference is minimized at $c=\mu_g$. In data, the group mean of observed values estimates that conditional mean, provided the observed cases are representative within the group. The derivation explains the mean's squared-error rationale; it does not show that the observed group mean is a reliable estimate for a group with very few observations.

Important limitation: Replacing several missing values with one group mean or median reduces the variation in the completed data and can understate uncertainty. It is useful as a transparent data-preparation choice in some predictive workflows, but it does not create new information about each missing value or automatically make statistical inference valid.

Reproducible Implementation in R

The following base R pattern uses a group median, requires at least 2 observed values per group, and falls back to the global observed median. It creates a separate completed variable and a missingness flag rather than overwriting the original values. Here, the grouping variable is called segment and the numeric variable is called amount.

R workflow:
dat$amount_was_missing = is.na(dat$amount)
observed = !is.na(dat$amount)
group_n = tapply(dat$amount[observed], dat$segment[observed], length)
group_median = tapply(dat$amount[observed], dat$segment[observed], median)
fallback = median(dat$amount, na.rm = TRUE)
filled = dat$amount
missing = is.na(dat$amount)
keys = as.character(dat$segment[missing])
candidate = unname(group_median[keys])
n_for_missing = unname(group_n[keys])
use_fallback = is.na(candidate) | is.na(n_for_missing) | 2 > n_for_missing
candidate[use_fallback] = fallback
filled[missing] = candidate
dat$amount_imputed = filled

In this code, a missing grouping value has no usable group summary, so it receives the fallback. A group with no observed amounts also has no usable median and receives the fallback. If the global observed median is itself unavailable, the workflow should stop for review instead of treating an undefined value as a valid imputation. In production, also check that the grouping variable is represented consistently and that the fallback is calculated only from the appropriate data partition.

For predictive modeling, compute group summaries and the fallback using the training data, then apply those fixed values to missing predictor values in validation or test data. Recalculating them using the full data would allow information from held-out observations to influence preprocessing. Also check that the grouping variable and its levels will be available when predictions are made. If a new group appears at prediction time, the documented fallback can be used; do not silently estimate a new summary from the held-out cases.

Common Mistakes and Checks

Using a group without a rationale. A finely split or arbitrary group can create unstable summaries. Explain why the group should be related to the imputed variable.
Ignoring group size. A mean or median based on one observed value is calculable, but that does not make it dependable. Apply a stated minimum-size rule when appropriate.
Calculating summaries from filled values. Reusing earlier imputations in later group summaries can feed replacements back into the estimates. Calculate summaries from the original observed values.
Leaving missing group labels unresolved. A case with no group label cannot receive a group-specific summary. Apply a defined fallback or investigate the missing grouping field.
Imputing before splitting the data. For prediction, estimate the summaries on training data and carry them forward. Do not use validation or test values to set the imputation rules.
Treating filled values as observed truth. Retain a missingness flag or other audit information so later analysis can distinguish recorded values from replacements.

Key takeaway: Group-specific imputation is defensible when the group has a substantive rationale and observed cases are informative about missing cases within that group. Specify the summary, minimum group size, fallback, and data partition before filling values; retain an audit trail and remember that single-value replacements do not preserve the original uncertainty or variability.

Check Your Understanding

For each problem, state the applicable group summary or fallback and check that the number of observed values meets the stated rule.

A numeric predictor has two missing values in group K, whose observed values are 40 and 50, and one missing value in group L, whose observed values are 70 and 90. Use group means with no minimum-size rule. What values are imputed? (Answer: 45 for group K and 80 for group L.)
Use a minimum group size of 3 and the global observed median as fallback. Group A has observed values 2 and 8; group B has observed values 10, 12, and 16. Each group has one missing value. What are the two replacements? (Answer: Group A uses the fallback 10; group B uses its group median 12.)
Use group means with a minimum group size of 2 and the global observed mean as fallback. Group X has observed values 3 and 9; group Y has one observed value, 15; group Z has no observed values. Each group has a missing value. Calculate all replacements. (Answer: The global mean is $(3+9+15)/3=9$; X receives 6, while Y and Z each receive 9.)
A case has a missing numeric value and a missing group label. The chosen group summary cannot be calculated for this case. Under a rule that uses the global observed median whenever a group summary is unavailable, what should be used, and what should be recorded? (Answer: Use the global observed median and record that the value was imputed and that the group label was unavailable.)
A model's training data have group medians of 20 for group A and 35 for group B, and a training-data global median of 28. A validation case has a missing predictor in a new group C. Under a rule that falls back for unseen groups, what value should be used, and should group C's validation values be used to recalculate it? (Answer: Use 28; do not recalculate from validation data.)

What You'll Learn