What Does an Observed Likelihood Tell Us?
When outcomes are missing, the likelihood is built from what was actually observed: the response pattern and the recorded values within that pattern. It can provide evidence about the distribution of those observable data. A different question is what the unobserved outcomes would have been. Answering that question may require restrictions on the complete-data distribution that are not established by the observed likelihood itself.
This distinction is easy to blur because a statistical model often uses one parameterization for both observable and unobservable features. A fitted model may report an estimate of a full-population mean, for example, but that estimate reflects both the likelihood and the model's assumptions. The likelihood's contribution must be separated from the assumptions that connect observed outcomes to missing ones.
The earlier material on observed-data distributions and identification targets makes the key distinction: the observable distribution and the complete-data distribution are not the same object. Here the practical question is how to read the likelihood so that an estimate based on observable information is not mistaken for a data-established conclusion about missing outcomes.
A Binary Outcome: What Appears in the Likelihood?
Let \(Y\) be a binary outcome, with \(Y=1\) denoting success, and let \(R=1\) mean that \(Y\) is observed. Assume independent, identically distributed subjects, and suppose \(R\) is recorded for everyone while \(Y\) is recorded only when \(R=1\). Define \(\pi=P(R=1)\), \(q=P(Y=1\mid R=1)\), and \(u=P(Y=1\mid R=0)\). The values of \(R\) and observed \(Y\) are observable; \(u\) describes outcomes among subjects whose outcomes are missing.
For one subject, the observable categories have probabilities \(\pi q\) for \(R=1,Y=1\), \(\pi(1-q)\) for \(R=1,Y=0\), and \(1-\pi\) for \(R=0\). The last category does not reveal the subject's value of \(Y\). If the counts in these categories are \(n_{11}\), \(n_{10}\), and \(n_0\), the observed-data likelihood is therefore:
The parameter \(u\) does not appear. For this unrestricted model, changing the outcome distribution among nonrespondents while holding \(\pi\) and \(q\) fixed leaves the probability of every possible observed data set unchanged. Thus the likelihood contains information about the response probability and the success probability among respondents, but does not separate possible values of the missing-outcome probability.
Worked Example: Reading the Likelihood for a Missing Binary Outcome
Hypothetical data. In a sample of 100 subjects, outcomes are observed for 40. Among those 40, 24 have \(Y=1\) and 16 have \(Y=0\); the other 60 outcomes are missing. Assume the sampling setup just stated, with no additional restrictions on \(u\).
There are \(n_{11}=24\), \(n_{10}=16\), and \(n_0=60\). The observed likelihood is
Maximizing the response part gives \(\hat{\pi}=40/100=0.4000\). Maximizing the respondent-outcome part gives \(\hat{q}=24/40=0.6000\). These estimates summarize the observed response proportion and the observed success proportion among respondents.
The full-population success probability, if it is the target, is
Substituting the observed estimates shows the extrapolation explicitly:
For instance, if \(u=0.2000\), the resulting value is \(0.2400+0.6000(0.2000)=0.3600\). If \(u=0.8000\), it is \(0.2400+0.6000(0.8000)=0.7200\). Both choices yield exactly the same observed likelihood. The two resulting population success probabilities are not two estimates selected by the observed likelihood; they illustrate how conclusions about the full population change when an unobserved-outcome assumption changes.
Why Equal Observed Likelihoods Matter
The central justification is direct. Suppose two complete-data models induce the same probability for every possible observable data outcome. For any realized observed data set \(o\), their observed-data likelihoods are the same, because each likelihood is the probability assigned to \(o\). No procedure that uses only this likelihood can choose between those models on the basis of the observed data. The models may nevertheless imply different distributions for outcomes that were not observed.
For each possible complete-data configuration, retain the response indicators and the outcomes that are observed. This induces a distribution for the observable data.
If two complete-data models give the same probability to every observable outcome, they are observationally equivalent for this data structure.
The observed likelihood can distinguish models only when their induced observable distributions differ. It cannot resolve differences confined to unobserved outcomes.
This argument does not say that an observed likelihood is uninformative. It says that its information is about the observable distribution, and that the model determines which parameters are linked to that distribution. If a parameterization has a flat likelihood in one direction, the data provide no likelihood-based distinction in that direction. If the likelihood varies only weakly, estimates may be imprecise even when the parameter is identifiable. Structural nonidentification and weak identification are different problems.
Strata Can Refine Observable Information
Observed information can be more detailed than a single response rate and respondent success proportion. Covariates or other observed variables may divide the data into strata, allowing the likelihood to describe response patterns and recorded outcomes separately within each stratum. This can sharpen what is learned about the observable distribution. It does not, by itself, reveal missing outcomes within those strata.
Worked Example: Observable Patterns in Two Strata
Hypothetical data. Suppose a sample contains 50 subjects in stratum A and 50 in stratum B. In A, 30 outcomes are observed and 18 of those are successes. In B, 10 are observed and 4 are successes. Assume independent sampling within the stated strata, with no restriction linking missing and observed outcomes.
The observed response proportions are \(30/50=0.6000\) in A and \(10/50=0.2000\) in B. The observed success proportions among respondents are \(18/30=0.6000\) in A and \(4/10=0.4000\) in B. These are distinct observable summaries, and a likelihood can use the counts in each stratum.
Let \(u_A\) and \(u_B\) be the success probabilities among nonrespondents in A and B, respectively. The observed data include 18 successes in A and 4 in B, or 22 observed successes in all. The other 60 subjects have missing outcomes. The expected total success proportion under specified values of \(u_A\) and \(u_B\) is represented by
The coefficients 20 and 40 are the numbers of missing outcomes in A and B. The observed counts identify the response and respondent-outcome summaries just calculated; they do not determine \(u_A\) or \(u_B\) in this unrestricted setup. Stratifying the observable likelihood has not made the missing outcomes observable.
When Assumptions Add a Conclusion
A model may impose a restriction that links missing outcomes to observed outcomes. For example, in the binary example, the assumption \(u=q\) says that success probability is the same among respondents and nonrespondents. Under that restriction, the full-population success probability is \(q\). This is a model-based conclusion: it follows from the restriction together with the observed estimate of \(q\), not from a likelihood that separately measured \(u\).
Worked Example: A Conclusion Conditional on an Assumption
Use the hypothetical sample of 100 subjects, with 40 observed outcomes and 24 observed successes. Suppose the analysis imposes \(u=q\), so the success probability is equal across response groups. The observed likelihood still estimates \(q\) by
Under the imposed restriction, the full-population success probability is \(q\), so its fitted value is \(0.6000\), or 60.00%. Equivalently, the restriction predicts \(60(0.6000)=36\) successes among the 60 nonrespondents; adding the 24 observed successes gives \(60\) out of \(100\). This arithmetic is consistent with the assumption, but the missing outcomes were not observed to check that prediction. In this example, the observed likelihood alone cannot distinguish \(u=q\) from other values of \(u\).
The distinction is not unique to one missingness framework. A pattern-mixture model describes outcome distributions by response pattern; a selection model describes response probabilities conditional on outcomes. Either can be useful for expressing assumptions. Their parameterizations do not change the observed records. A restriction in either representation may produce a full-data conclusion, but the restriction should not be described as evidence extracted from the observed likelihood unless it has observable implications that the data can assess.
Comparisons, Diagnostics, and Limitations
| Approach | What it uses | Interpretive limit |
|---|---|---|
| Unrestricted observed-data likelihood | Probabilities of response patterns and recorded outcomes | Cannot distinguish complete-data models with the same observable distribution |
| Likelihood under a restricted missing-data model | The observed likelihood plus restrictions linking observed and missing outcomes | Full-data conclusions depend on restrictions as well as on the data |
| Complete-data likelihood | Probabilities for outcomes and response indicators if all outcomes were known | Missing outcomes are not available to evaluate it directly; the observed likelihood must account for them |
Useful diagnostics begin with the observable part of the model. Check whether fitted response-pattern probabilities and the distributions of recorded outcomes are compatible with the data. Look for sparse or empty observable patterns, numerical instability, or a likelihood that is nearly flat over relevant parameter combinations. These can signal poor precision or failure to estimate even some observable features reliably. But a good fit to observable data cannot, by itself, verify an assumption about unobserved outcomes when competing assumptions yield the same observable distribution.
A common mistake is to say, “The likelihood fitted well, so the estimated missing-outcome distribution must be correct.” The reasoning fails because a fit assessment checks the model's implications for observed data. If alternative missing-outcome distributions make identical observable predictions, the fit cannot discriminate among them. The correction is to report separately (1) the observable summaries or parameters supported by the likelihood and (2) the assumptions required for any extrapolation to unobserved outcomes.
The observed likelihood is also not a guarantee of precise estimates: a finite sample can leave substantial uncertainty about identifiable observable parameters. Conversely, a parameter may be reported with a narrow interval under a restrictive model while the conclusion remains sensitive to an untestable restriction. The interpretation must therefore name both the target and the assumptions under which it is being estimated.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the binary setup, which parameters appear in the unrestricted observed-data likelihood, and which parameter does not?
- A hypothetical sample has 80 subjects, 30 observed outcomes, and 12 observed successes. Calculate the response proportion and the success proportion among respondents.
- For the sample in question 2, express the fitted full-population success proportion as a function of \(u\), the success probability among nonrespondents.
- Can two complete-data models have different missing-outcome probabilities but identical observed-data likelihoods? State the condition that makes this possible.
- If an analysis imposes \(u=q\), is the resulting full-population conclusion established by the observed likelihood alone? Explain.
Solutions. (1) The likelihood contains \(\pi\) and \(q\); it does not contain \(u\), because the outcomes for \(R=0\) are not observed. (2) The response proportion is \(30/80=0.3750\), and the respondent success proportion is \(12/30=0.4000\). (3) The fitted proportion is \((0.3750)(0.4000)+(0.6250)u=0.1500+0.6250u\). (4) Yes. If the models induce the same probability for every observable data outcome, their observed-data likelihoods are identical even if their missing-outcome probabilities differ. (5) No. The value follows from the restriction \(u=q\) plus the observed-data estimate of \(q\); the unrestricted observed likelihood does not distinguish that restriction from other values of \(u\).