Why Sparsity Matters for Delta Adjustment
A delta adjustment changes the assumed outcomes for people whose outcomes are missing. But it does not create information about the reference outcomes those people would have had. When few comparable outcomes are observed, the reference predictions may be uncertain; when the missing cases have covariate values rarely or never seen among observed cases, predictions may depend mainly on the model's extrapolation. In either situation, a carefully chosen delta can still produce a precise-looking calculation built on a weak reference.
The key is to assess two things separately: whether the observed data support the reference prediction, and what departure from that reference is represented by delta. The observed data can help assess the first; they generally cannot determine the second. This tutorial focuses on how sparse observations and weak overlap affect the first part.
This definition concerns conditional means, not necessarily individual missing outcomes or their full distribution. The reference model must be adequate for the covariate pattern being predicted, and the delta value must be meaningful on the outcome scale. A fixed additive delta leaves the reference prediction's sampling uncertainty unchanged: it shifts the prediction and its interval by the same amount. It does not account for uncertainty about the selected delta.
Two Different Weaknesses in the Reference Prediction
Sparsity means that few observed outcomes inform the prediction, even when the relevant covariate pattern is represented. Estimates based on small samples tend to be variable. Weak overlap means that covariate patterns among cases with missing outcomes have little representation among cases with observed outcomes. A fitted model may then be asked to predict in a region with little or no observed support. These problems can occur together, but they are not interchangeable: more observations in one region do not necessarily help predict in another.
A useful target-specific summary is the distribution of \(X\) among the cases whose outcomes are missing, compared with the distribution among observed cases. For continuous covariates, inspect ranges, density plots, and the location of missing cases relative to observed cases. For categorical covariates, tabulate observed and missing counts in each category and relevant combination of categories. A large overall observed sample can conceal an empty or tiny cell for an important subgroup.
When the reference model is estimated from a small but representative sample, uncertainty can sometimes be quantified under explicit sampling and model assumptions. When a target covariate value lies outside observed support, a model-based standard error alone is not a complete measure of uncertainty: it is conditional on the chosen model being correct in that unsupported region. The data cannot directly check that extrapolation.
Worked Example: A Sparse Observed Stratum
Suppose, hypothetically, that four observed outcomes in a clinically relevant stratum are 5, 6, 7, and 8 points. Three outcomes in that stratum are missing. Assume the four observed outcomes are independent and identically distributed from a normal distribution within this stratum, and use their mean as the reference prediction. These assumptions make the usual one-sample \(t\) interval appropriate for the stratum mean.
The observed mean is \(\bar{x}=(5+6+7+8)/4=6.5\) points. The sample standard deviation is \(s=\sqrt{[(5-6.5)^2+(6-6.5)^2+(7-6.5)^2+(8-6.5)^2]/3}=\sqrt{5/3}=1.291\) points. Thus the estimated standard error of the reference mean is \(SE=s/\sqrt{n}=1.291/\sqrt{4}=0.6455\) points.
For illustration, set the additive delta to \(-2\) points. The reference prediction for each missing case is 6.5 points, and the delta-adjusted conditional mean is \(6.5-2=4.5\) points. A 95% \(t\) interval for the reference mean, with \(df=4-1=3\), is \(6.5\pm3.182(0.6455)\), or approximately 4.446 to 8.554 points. Shifting both endpoints by \(-2\) gives 2.446 to 6.554 points for the delta-adjusted mean, treating delta as fixed.
The interval is wide because only four outcomes inform the reference mean. It is an interval for the conditional mean under the stated normal, independent, identically distributed sampling assumption; it is not an interval for each person's realized outcome. If the four outcomes were independent but had different variances or distributions, the calculation \(s^2/n\) would not be justified by independence alone.
Overlap and Extrapolation
A regression model can produce a numerical prediction even where observations are scarce. Numerical availability is not evidence that the data support the prediction. In particular, a model's standard error for a fitted mean usually reflects sampling variation under that model; it does not automatically include error from an incorrect functional form or a relationship that changes outside the observed range.
Worked Example: A Linear Prediction Beyond the Observed Range
Consider hypothetical observed data with covariate values 0 and 1 only: at \(X=0\), the outcomes are 4, 5, and 6; at \(X=1\), they are 6, 7, and 8. Suppose missing outcomes of interest occur at \(X=2\). Fit a straight-line model with independent, normally distributed errors of constant variance. The model is an assumption; the data at 0 and 1 cannot verify that the linear relationship continues at 2.
The two group means are 5 and 7, so the fitted line is \(\widehat m_0(x)=5+2x\). At \(x=2\), the reference prediction is \(5+2(2)=9\) points. The residual sum of squares is 4: within each of the two groups, the residuals are \(-1,0,1\), whose squared values sum to 2. With \(6-2=4\) residual degrees of freedom, the estimated error variance is \(MSE=4/4=1\) square point.
For a fitted mean at \(x=2\), the leverage factor is \(1/6+(2-0.5)^2/1.5=1.667\), since \(\bar{x}=0.5\) and \(S_{xx}=1.5\). The estimated standard error is therefore \(\sqrt{1(1.667)}=1.291\) points. With \(df=4\), a 95% interval for the fitted mean is \(9\pm2.776(1.291)\), or approximately 5.416 to 12.584 points.
For a fixed delta of \(-1\) point, the adjusted mean is \(9-1=8\) points and the shifted interval is 4.416 to 11.584 points. These calculations are conditional on the straight-line, constant-variance model. Because no observed case has \(X=2\), the interval does not protect against a nonlinear relationship at 2 or another failure of extrapolation. The delta shift does not repair that limitation.
How Sparsity Can Dominate a Marginal Estimate
A sparse stratum can matter greatly when it represents a substantial share of the target population. The following calculation uses a fixed target composition across two strata. It estimates the target mean by applying each stratum's observed mean to everyone in that stratum and adding the delta contribution for the missing outcomes. The variance calculation requires independent, identically distributed outcomes within each stratum and independent samples between strata; without such assumptions, the displayed variance estimator is not warranted.
Worked Example: A Large Missing Subgroup With Two Observed Outcomes
In a hypothetical target population, stratum A has 22 people: 20 observed outcomes with mean 10 and sample standard deviation 4, and 2 missing outcomes. Stratum B has 20 people: 2 observed outcomes, 18 and 22, and 18 missing outcomes. The two observed values in B have mean 20 and sample standard deviation \(\sqrt{8}=2.828\). Assume independent, identically distributed outcomes within each stratum, with independent sampling between strata.
The reference estimate for the target mean, using the observed stratum means for all people in their respective strata, is \([22(10)+20(20)]/42=620/42=14.76\) outcome units. Set delta to \(-3\) units for missing outcomes in both strata. There are 20 missing outcomes in total, so the delta-adjusted estimate is \(14.76+(20/42)(-3)=13.33\) units.
The standard error of this stratum-weighted reference estimate can be calculated from the independent sample means. The stratum weights are \(22/42\) and \(20/42\), so the estimated variance is \((22/42)^2(4^2/20)+(20/42)^2(2.828^2/2)=0.2195+0.9070=1.1265\) square units. The standard error is \(\sqrt{1.1265}=1.061\) units. A fixed delta shifts the estimate but not this variance calculation.
Although stratum B contains fewer than half of the target population, its 18 missing outcomes receive a reference mean estimated from only two observed outcomes. Its contribution to the estimated variance, 0.9070, is much larger than stratum A's 0.2195. The arithmetic does not resolve possible model failure in B; it shows how a sparse observed stratum can strongly influence the result. With only two observations, distributional assumptions and the sample standard deviation itself are also particularly fragile.
Diagnostics and Practical Alternatives
Before interpreting a delta-adjusted estimate, identify which observed cases inform each reference prediction. Check stratum counts and covariate distributions, and inspect whether predictions for missing cases lie inside the observed covariate range. Compare predictions across reasonable model specifications, particularly where data are sparse. Large changes are a warning that the result depends on modeling choices. Implausible predictions are another warning, but plausible-looking predictions do not establish that extrapolation is reliable.
A useful distinction is between a reference model that is uncertain because its parameters are estimated imprecisely and a reference model whose behavior in the target region is unsupported. Standard errors or intervals can communicate the first kind of uncertainty under their assumptions. Sensitivity to functional form, influential observations, or restriction of the target population can expose the second, but no diagnostic based only on observed outcomes can verify an unsupported relationship.
| Approach | What it can address | Important limitation |
|---|---|---|
| Delta adjustment using the existing reference model | Represents a specified departure from the observed-outcome reference mean. | Does not increase the information supporting a sparse or extrapolated reference prediction. |
| Pooling sparse strata | Can stabilize estimates when a shared mean or regression structure is reasonable. | May conceal meaningful subgroup differences; pooling is a modeling assumption, not new data. |
| Restricting the target to supported covariate regions | Reduces dependence on predictions in regions without observed support. | Changes the target population and may not answer the original question. |
| External information | May inform reference predictions in poorly represented regions if it is relevant and comparable. | Differences in population, measurement, or setting can limit its applicability. |
None of these choices is universally preferable. Pooling trades local flexibility for assumptions shared across cases; restricting the target trades generality for better support; external information requires a defensible connection to the target setting. Delta adjustment addresses a different question: how missing outcomes may depart from the reference. These choices should not be treated as substitutes for one another.
Common Mistakes and Limits
- Confusing a fixed delta with certainty in the reference. A specified shift does not make a sparse reference estimate precise. Report or examine reference uncertainty separately.
- Treating a model prediction as observed support. Software can return a prediction outside the data range, but its value depends on the assumed model there.
- Using independence alone to justify \(s^2/n\). The usual sample-variance estimate for the mean's variance requires an appropriate sampling model, such as independent, identically distributed outcomes within the stratum. Under independent outcomes with heterogeneous variances, the target variance is \(\sum_i\operatorname{Var}(Y_i)/n^2\), which generally is not estimated by \(s^2/n\).
- Assuming a large total sample guarantees adequate overlap. Counts must be examined in the covariate regions that inform the predictions of interest.
- Calling a result data-driven when it is extrapolation-driven. If reasonable model changes produce materially different predictions in unsupported regions, state that dependence rather than presenting one model's output as established by the data.
The central limitation is fundamental: the observed outcomes can assess the fit of a reference model where data exist, but cannot directly establish its conditional mean where comparable observations do not exist. A delta analysis can make assumptions about missing outcomes explicit; it cannot turn an unsupported reference prediction into an empirically verified one.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A stratum has two observed outcomes, and a model predicts missing outcomes in that stratum. Name two distinct reasons the reference predictions may be weakly supported.
- If a fixed additive delta is changed from \(-2\) to \(-4\), what happens to a reference-mean interval under the same model?
- Why does a small regression standard error not prove that an extrapolated prediction is reliable?
- In the two-stratum example, which stratum contributes more to the estimated variance, and why?
- What assumption is needed for estimating the variance of a stratum sample mean by \(s^2/n\) using the usual sample variance?
Solutions. (1) The sample may be too small to estimate the stratum mean precisely, and the missing cases may have covariate patterns poorly represented among observed cases. Either problem can occur without the other. (2) Both interval endpoints shift down by 2 outcome units, while the interval width is unchanged if delta is treated as fixed and the reference model is unchanged. (3) The standard error reflects sampling uncertainty conditional on the fitted model; it does not generally include bias from an incorrect relationship beyond the observed range. (4) Stratum B contributes more: its variance contribution is 0.9070 versus 0.2195 for A, because its mean is based on only two observed outcomes despite its large target weight. (5) A standard sufficient assumption is independent, identically distributed outcomes within the stratum; independence alone does not justify that estimator when outcomes may have different variances.