What Can Be Learned From Outcomes That Are Observed?
A delta adjustment describes how the distribution of an unobserved outcome may depart from a specified reference distribution. Observed outcomes can help estimate that reference distribution and show whether its observable implications are plausible. They do not, by themselves, reveal how the unobserved outcomes differ from the reference. Keeping those two roles separate is essential: a well-fitting reference model does not make its delta empirically known.
Let \(Y\) be an outcome, \(X\) a set of observed covariates, and \(R\) an indicator equal to 1 when \(Y\) is observed and 0 when it is missing. The observed data include \(X\), \(R\), and \(Y\) only for cases with \(R=1\). A reference model specifies a distribution for the missing outcomes using information available from observed cases and covariates. For a simple example, the reference mean for a case with covariates \(X=x\) might be the mean among observed cases with those covariates.
This definition describes a mean shift only. A full distributional shift requires specifying how the distribution changes, not just its mean. The central identification point applies either way: the observed data contain no recorded values of \(Y\) for \(R=0\), so they cannot directly reveal that missing-outcome distribution.
Why Delta Is Not Identified by the Observed Data
The point can be seen from the observed-data likelihood. Write \(\pi(x)=P(R=1\mid X=x)\), and let \(f_1(y\mid x)\) be the outcome density among cases with \(R=1\). For a case with \(R=1\), the observed contribution involves \(\pi(x)f_1(y\mid x)\). For a case with \(R=0\), the outcome is absent, so the contribution involves \(1-\pi(x)\), not a density evaluated at a missing \(y\).
This expression leaves out the conditional distribution of \(Y\) among cases with \(R=0\). Consequently, if that distribution is allowed to vary freely, changing it does not change the observed-data likelihood. A delta parameter that indexes changes to that distribution is therefore not identified by these observed data alone. This is a statement about the information in the data, not about whether an estimate would be precise: even an arbitrarily large sample from the same observed-data process cannot distinguish delta values without further restrictions or information.
The conclusion changes only if additional information or assumptions link the missing outcomes to something observed. For example, a reference assumption may set the missing distribution equal to a distribution predicted from observed cases, and an external validation sample may measure outcomes that are missing in the main study. Such assumptions or data can determine, restrict, or inform a delta. They are not evidence supplied by the unobserved outcomes in the main study itself.
The observed cases contribute outcome values to the likelihood; the missing cases contribute their observed covariates and missingness indicators, but no outcome values.
The values \(Y\) for \(R=1\), together with \(X\), can inform the observed outcome distribution and parameters of a chosen reference model.
Delta changes the assumed distribution for \(Y\) when \(R=0\), which does not appear in the observed-data likelihood above.
If a value is selected or bounded, say whether this comes from an assumption, external evidence, or a structural restriction. Do not describe it as estimated from the missing outcomes in the study.
Worked Example: The Same Observed Data, Different Delta Values
Suppose a hypothetical study has 100 participants: 60 have an observed continuous outcome, with sample mean \(\bar{x}=10.0\) points, and 40 have a missing outcome. Assume for this illustration that the reference model assigns the missing cases a mean of 10.0 points. Consider an additive shift \(\delta\).
Under this setup, the assumed mean for the missing cases is \(10.0+\delta\) points. The implied mean across all 100 participants is
For \(\delta=0\), the implied overall mean is \(10.0+0.40(0)=10.0\) points. For \(\delta=-2.0\) points, it is \(10.0+0.40(-2.0)=9.20\) points. For \(\delta=1.5\) points, it is \(10.0+0.40(1.5)=10.6\) points.
All three calculations use the same 60 observed outcomes and the same response pattern. The different overall means result from different assumptions about the 40 missing outcomes; the observed outcomes do not select among those delta values. The arithmetic also shows why an unidentified delta can matter: in this example, each one-point change in delta changes the implied overall mean by 0.40 points.
The illustration assumes that the reference mean of 10.0 points is appropriate for the missing cases and that the same additive delta applies to all of them. Those are modeling choices. The observed mean among the 60 participants does not establish that the other 40 would have had that mean under the reference model.
What Observed Outcomes Can Diagnose
Although observed data do not identify delta, they are informative. They show the outcome distribution for cases with \(R=1\), and the covariates and response pattern for both observed and missing cases. If the reference model uses covariates, observed outcomes can inform how the outcome varies with those covariates among observed cases. The data can also show whether the study contains observed cases across the covariate values where predictions for missing cases are needed. These are empirical features relevant to the reference model; they do not disclose the unobserved outcomes.
An empirical check can reveal a mismatch in observable data. For instance, a reference-model prediction that systematically misses observed outcomes in a group where outcomes are available is a warning about that model’s observable fit. By contrast, a claim that the missing cases would have followed the same conditional outcome distribution as the observed cases is an assumption about unobserved values. Agreement with observed data can make a reference model more credible for its intended role, but cannot verify that assumption or identify the departure from it.
Worked Example: Covariates Inform the Reference, Not the Delta
Consider a hypothetical study with a binary covariate \(X\). Among participants with \(X=0\), 30 outcomes are observed with mean 5.0 points and 10 are missing. Among those with \(X=1\), 20 outcomes are observed with mean 8.0 points and 40 are missing. Suppose the reference model uses the observed mean within each covariate group as its reference mean.
The observed data inform two different reference means: 5.0 points for \(X=0\) and 8.0 points for \(X=1\). Applying those means to the missing cases gives an assumed missing-outcome total of
The observed-outcome total is \(30(5.0)+20(8.0)=150+160=310\) points. There are 100 participants altogether, so under this reference model the implied overall mean is \((310+370)/100=6.80\) points.
Now suppose a common additive delta of \(-1.0\) point is applied to all 50 missing outcomes. Their assumed total becomes \(370+50(-1.0)=320\) points, and the implied overall mean is \((310+320)/100=6.30\) points. The covariate-specific observed means help construct the reference prediction and account for the different covariate composition of the missing group. They do not tell us whether the appropriate delta is 0, \(-1.0\), or another value. The calculation also relies on having an adequate reference model and on choosing a common shift; neither choice is proved by the arithmetic.
Empirical Checks and Untestable Assumptions
A useful distinction is between statements that can be checked against observed data and statements about the outcomes that remain missing. The first category concerns observable quantities or observable implications of a model. The second requires assumptions, external information, or both.
| Question | What the data can contribute | What remains unestablished |
|---|---|---|
| Are outcomes associated with measured covariates among observed cases? | Observed outcomes and covariates can estimate and describe that association. | Whether the same association holds among cases whose outcomes are missing. |
| Are there observed cases at covariate values represented among missing cases? | The covariate values and response indicators show where observed data are available or sparse. | Whether predictions for poorly represented values are correct. |
| Does a reference model reproduce observable features used to fit or assess it? | Observable discrepancies can challenge the model’s fit to those features. | Good observable fit does not verify its assumptions for missing outcomes. |
| Do missing outcomes differ from the reference distribution by delta? | The missingness pattern and observed outcomes provide context for sensitivity analysis. | The missing-outcome departure itself is not observed and is not identified from those outcomes alone. |
These distinctions matter when reporting conclusions. It is appropriate to say that observed data support a particular reference-model relationship among observed cases, or that predictions are uncertain in covariate regions with little observed information. It is not appropriate to say that a good fit proves the missing outcomes follow the reference distribution, or that the study estimated delta when delta was specified for sensitivity analysis.
Worked Example: Classifying Evidence in a Hypothetical Study
A hypothetical study records a 0-to-20-point symptom score and a treatment indicator. Scores are observed for 70% of the treatment group and 85% of the comparison group. Among participants with observed scores, the treatment-group mean is 12.0 points and the comparison-group mean is 11.0 points. A researcher proposes that missing scores average 1.5 points below their reference predictions.
Empirically available: The response proportions are 0.70 and 0.85, respectively. The observed mean difference is \(12.0-11.0=1.0\) point, comparing observed treatment and comparison participants. These facts describe the observed data. They do not establish a treatment effect for all participants, since the outcomes for those with missing scores are not included in these means.
Not empirically determined by these summaries: The proposed \(-1.5\)-point shift concerns missing scores relative to reference predictions. No missing score is available to calculate that difference. The value can be examined as a sensitivity assumption, but the response proportions and observed means do not estimate it.
What would add information: A separate source that measured outcomes for a relevant group of otherwise-missing participants could help assess the shift, subject to the relevance of that source. Alternatively, a scientific restriction could constrain the missing distribution. Without such information or restrictions, choosing \(-1.5\) points is a transparent assumption, not an empirical finding from these observed means.
Common Pitfalls and Limits
- Calling a plausible delta “estimated.” A clinically reasonable value may be useful in a sensitivity analysis, but plausibility is not identification. State how the value was chosen.
- Treating a good reference-model fit as proof about missing cases. Fit checks evaluate observable features. They cannot directly compare predictions with outcomes that were never recorded.
- Confusing a response-rate difference with a delta estimate. Response rates describe missingness. Without additional assumptions, they do not measure the outcomes among missing cases.
- Overlooking weak covariate support. If some covariate values among missing cases have few or no observed counterparts, the reference prediction there relies heavily on extrapolation. More data in well-represented regions do not automatically solve that problem.
- Assuming every delta is unrestricted. External data, a known measurement process, or a defensible structural restriction may constrain the missing distribution. In that case, be explicit about which source supplies the constraint; it is not information extracted from missing outcome values in the main study.
Observed data therefore play a specific, valuable role: they inform the distribution used as a reference and can expose observable weaknesses or gaps. Sensitivity analysis addresses what those data cannot settle by examining how conclusions change under stated alternatives for the missing outcomes. Its results should be interpreted conditionally on the reference model and the delta values considered, rather than as a data-driven estimate of the missing-outcome departure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why can two different delta values produce the same observed-data likelihood?
- A reference model gives a missing-case mean of 20 units and an additive delta is \(-3\) units. What is the assumed missing-case mean, and which part of this calculation is not learned directly from the missing outcomes?
- In a study, observed outcomes are well represented across the covariate values found among missing cases. What does this support, and what does it not establish?
- Would a larger sample of the same observed-data process necessarily identify delta? Explain.
Solutions. (1) The observed-data likelihood includes outcome values for \(R=1\), but for \(R=0\) it includes the response pattern and covariates without an outcome density. Changing the missing-outcome distribution indexed by delta therefore leaves that likelihood unchanged. (2) The assumed mean is \(20+(-3)=17\) units. The \(-3\)-unit departure is an assumption or external-information-based choice, not a value calculated from outcomes that are missing. (3) It supports having observed data in relevant covariate regions for estimating the reference relationship, though prediction still depends on the reference model. It does not establish that missing outcomes follow that relationship or identify their delta. (4) No. A larger sample can improve estimation of observable quantities, but if the observed-data likelihood still contains no information about the missing-outcome distribution, delta remains unidentified without added assumptions or information.