A Binary-Outcome Proof of MNAR Nonidentification

Why the Observed Data May Not Determine the Outcome Distribution

When an outcome is missing, its value is not recorded, but the fact that it is missing is observed. That distinction matters: the observed response rate tells us how often outcomes are recorded, and the observed outcomes tell us their distribution among respondents. Neither fact alone reveals how the missing outcomes are distributed. If response may depend on the outcome itself, different complete-data populations can agree on everything observed and disagree about the population event probability.

A binary outcome makes this issue especially clear. We can write down every possible full-data distribution using just one unknown probability for the missing outcomes, then check whether the observed data restrict that probability. This gives a direct proof of nonidentification: two different values of the target can generate exactly the same observed-data distribution.

Definition: Let \(Y\in\{0,1\}\) be a binary outcome, and let \(R\in\{0,1\}\) indicate whether it is observed: \(R=1\) means observed and \(R=0\) means missing. The observed data consist of \(R\), and of \(Y\) when \(R=1\). Write \(\rho=P(R=1)\) for the response rate, \(\theta=P(Y=1\mid R=1)\) for the event probability among respondents, and \(\eta=P(Y=1\mid R=0)\) for the event probability among nonrespondents. The population event probability is \(\pi=P(Y=1)\).

Assume first that \(0<\rho<1\), so there are both respondents and nonrespondents in the population, and that the observed outcome distribution is defined. The observed-data distribution identifies \(\rho\) and \(\theta\): it gives the probability of response and, among those responding, the probability of an event. It does not directly reveal \(\eta\), because \(Y\) is unobserved when \(R=0\). The question is whether the observed quantities nevertheless determine \(\eta\) indirectly. The following construction shows that, without a further restriction on response, they do not.

A Family of Complete-Data Laws With the Same Observed Data

For any proposed value \(\eta\) between 0 and 1, define the joint probabilities of response and outcome by combining the conditional outcome probabilities within each response group:

\begin{aligned} P(R=1,Y=1)&=\rho\theta, & P(R=1,Y=0)&=\rho(1-\theta),\\ P(R=0,Y=1)&=(1-\rho)\eta, & P(R=0,Y=0)&=(1-\rho)(1-\eta). \end{aligned}

These four probabilities are nonnegative and sum to 1. For every choice of \(\eta\), the two probabilities in the respondent group stay fixed. The observed data therefore have the same response rate \(\rho\), the same event probability \(\theta\) among respondents, and the same probabilities for every observable outcome-and-response combination. Only the split of the nonrespondents between \(Y=1\) and \(Y=0\) changes. Since that split is not observed, the observed distribution does not select one value of \(\eta\) from the full range \([0,1]\).

Theorem: If \(0<\rho<1\) and the only restrictions are that \(Y\) is binary and response may depend on \(Y\), then every \(\eta\in[0,1]\) is compatible with the same observed-data distribution. The population event probability consequently satisfies \(\pi=\rho\theta+(1-\rho)\eta\), so its compatible range is \([\rho\theta,\ \rho\theta+(1-\rho)]\). In particular, \(\eta\) and \(\pi\) are not identified by the observed data alone.

The formula for \(\pi\) follows by adding the event probabilities in the two disjoint response groups:

\begin{aligned} \pi &=P(Y=1)\\ &=P(Y=1,R=1)+P(Y=1,R=0)\\ &=\rho\theta+(1-\rho)\eta. \end{aligned}

As \(\eta\) varies from 0 to 1, this expression increases linearly from \(\rho\theta\) to \(\rho\theta+(1-\rho)\). Both endpoints are attainable under the construction: at \(\eta=0\), none of the nonrespondents have the event; at \(\eta=1\), all of them do. Every value between those endpoints is attainable too. This is an identification statement, not a claim about sampling uncertainty: even if the observed-data probabilities were known exactly, this family of different population event probabilities would remain.

Worked Example: Fixing the Observed Data

Suppose a hypothetical population has response rate \(\rho=0.60\), and 30% of respondents have the event, so \(\theta=0.30\). Find the observed-data probabilities and show how one missing-outcome distribution can complete them.

The observed probabilities are

P(R=1,Y=1)=0.60(0.30)=0.18,\qquad P(R=1,Y=0)=0.60(0.70)=0.42.

Thus 60% respond, with 18% of the whole population both responding and having the event and 42% responding without the event. Now suppose, solely as one possible completion, that \(\eta=0.10\). The nonrespondent probabilities are

P(R=0,Y=1)=0.40(0.10)=0.04,\qquad P(R=0,Y=0)=0.40(0.90)=0.36.

The four probabilities sum to \(0.18+0.42+0.04+0.36=1.00\). The population event probability in this completion is \(\pi=0.18+0.04=0.22\). The observed data determine the first two probabilities, but they do not establish that \(\eta=0.10\) or that \(\pi=0.22\).

Worked Example: Calculating the Compatible Population Range

Use the same hypothetical observed quantities, \(\rho=0.60\) and \(\theta=0.30\), but do not assume a particular value of \(\eta\). Calculate the endpoints of the compatible range for \(\pi\), then evaluate two possible completions.

The lower endpoint occurs at \(\eta=0\), and the upper endpoint at \(\eta=1\):

\begin{aligned} \pi_{\min}&=0.60(0.30)+0.40(0)=0.18,\\ \pi_{\max}&=0.60(0.30)+0.40(1)=0.58. \end{aligned}

So the population event probability can be anywhere from 0.18 to 0.58 under the stated assumptions. For example, if \(\eta=0.50\), then \(\pi=0.18+0.40(0.50)=0.38\). If \(\eta=0.90\), then \(\pi=0.18+0.40(0.90)=0.54\). Both choices retain exactly the same response rate and respondent event probability; they differ only in unobserved outcomes. This range describes compatibility with the observed distribution, not a confidence interval.

Showing That the Compatible Laws Can Be MNAR

A response mechanism is outcome-dependent when the probability of response differs according to \(Y\). In the constructed law, for nonzero conditioning probabilities,

P(R=1\mid Y=1) =\frac{\rho\theta}{\rho\theta+(1-\rho)\eta}, \qquad P(R=1\mid Y=0) =\frac{\rho(1-\theta)} {\rho(1-\theta)+(1-\rho)(1-\eta)}.

These conditional probabilities are calculated from the joint probabilities, not specified independently of them. In the nondegenerate case, they are equal precisely when the outcome distribution is the same among respondents and nonrespondents, \(\theta=\eta\). When \(\eta\ne\theta\), the outcome distribution varies with response status, and the response mechanism depends on the outcome in this marginal binary setting. The same observed data can therefore be compatible both with an outcome-independent completion and with outcome-dependent, or MNAR, completions.

Worked Example: Two Different Selection Mechanisms

Return to the hypothetical observed data \(\rho=0.60\), \(\theta=0.30\). Compare completions with \(\eta=0.10\) and \(\eta=0.50\). For each, compute the event probabilities and the response probability within each outcome group.

With \(\eta=0.10\), the joint probabilities are \(P(R=1,Y=1)=0.18\), \(P(R=0,Y=1)=0.04\), \(P(R=1,Y=0)=0.42\), and \(P(R=0,Y=0)=0.36\). Therefore

P(R=1\mid Y=1)=\frac{0.18}{0.18+0.04}=0.8182,\qquad P(R=1\mid Y=0)=\frac{0.42}{0.42+0.36}=0.5385.

Response is more likely for those with the event in this completion. Its population event probability is \(0.18+0.04=0.22\).

With \(\eta=0.50\), the joint probabilities are \(0.18,\ 0.20,\ 0.42,\) and \(0.20\), in the same order. Thus

P(R=1\mid Y=1)=\frac{0.18}{0.18+0.20}=0.4737,\qquad P(R=1\mid Y=0)=\frac{0.42}{0.42+0.20}=0.6774.

Now response is less likely for those with the event, and the population event probability is \(0.18+0.20=0.38\). Both mechanisms reproduce the observed probabilities 0.18 and 0.42 exactly, despite having different dependence directions and different population event probabilities. No observed-data comparison between these two completions can choose which missing-outcome distribution is correct.

Assumptions, Edge Cases, and What the Result Does Not Say

The construction assumes a binary outcome, a well-defined response indicator, a positive response rate so that \(\theta\) is available, and no restriction linking the missing-outcome distribution to the observed one. It does not assume random sampling or give a finite-sample estimator. The result concerns the probability distribution that could have generated the observations. Adding a substantive restriction on \(\eta\), such as \(\eta=\theta\), can narrow the compatible set, but that restriction is not supplied by the observed distribution itself.

The edge cases clarify why the theorem assumes \(0<\rho<1\). If \(\rho=1\), there are no missing outcomes and \(\pi=\theta\); the missing-group probability \(\eta\) is irrelevant. If \(\rho=0\), there are no respondents, so \(\theta\) is undefined and the observed data reveal no outcome distribution. At intermediate response rates, the width of the compatible range for \(\pi\) is \(1-\rho\): fewer observed outcomes leave a wider range, though the particular observed event rate also determines where that range lies.

This argument differs from weak identification. Weak identification concerns a parameter that is theoretically determined by the observable distribution but only imprecisely constrained in finite samples or under a nearly flat likelihood. Here, even exact knowledge of the observed distribution leaves multiple values of \(\eta\) and \(\pi\) possible. More data of the same observed form can estimate \(\rho\) and \(\theta\) more precisely, but cannot by themselves distinguish the compatible completions.

Diagnostics and Common Pitfalls

A useful diagnostic is to ask whether the observed-data probabilities contain any term involving \(\eta\). In this setup they do not: \(P(R=1,Y=1)=\rho\theta\) and \(P(R=1,Y=0)=\rho(1-\theta)\). If a proposed argument claims to estimate \(\eta\) from these probabilities alone, identify the equation that supplies that information. Without an additional assumption, no such equation is available.

Mistake: treating the respondent event proportion as the population event proportion. The equality \(\pi=\theta\) requires a restriction such as \(\eta=\theta\) when both response groups exist. The observed respondent proportion alone does not establish that equality.
Mistake: treating a fitted model as evidence about unobserved outcomes. A model can reproduce the observed response rate and respondent outcomes while relying on a chosen, untestable extrapolation for \(\eta\). Agreement with those observed quantities does not verify that extrapolation.
Mistake: confusing a range of compatible values with a confidence interval. The endpoints here arise from allowing every logically possible value of the missing-group event probability. Sampling uncertainty is a separate issue.
Mistake: concluding that all missingness must be MNAR. The construction shows that MNAR completions are possible and observationally equivalent to alternatives. It does not prove which mechanism generated any particular data set.

The main alternatives impose different targets or restrictions. A complete-case analysis describes respondents unless additional assumptions justify extending its conclusions to the full population. An MAR analysis restricts the relationship between response and unobserved outcomes, while a selection-model analysis specifies response probabilities conditional on outcomes. Such approaches can yield a point estimate only through their assumptions or model structure; the observed-data equivalence shown here is why those assumptions cannot generally be confirmed from the observed distribution alone. This tutorial’s construction does not assess which restriction is scientifically appropriate.

Key takeaway: With a binary outcome, response rate \(\rho\), and observed respondent event probability \(\theta\), the observed distribution leaves \(\eta=P(Y=1\mid R=0)\) unrestricted on \([0,1]\) when no additional assumptions are imposed. The population event probability is then \(\pi=\rho\theta+(1-\rho)\eta\), so distinct complete-data laws—including MNAR laws—can yield the same observed data.

Check Your Understanding

Answer each question using the definitions and construction above. Solutions follow the questions.

Suppose \(\rho=0.75\) and \(\theta=0.20\). What values of \(\eta\) are compatible, and what range of \(\pi\) follows?
For \(\rho=0.50\), \(\theta=0.40\), compare \(\pi\) under \(\eta=0.40\) and \(\eta=0.80\). Do these completions alter the observed-data distribution?
Why does learning the response rate and respondent event probability with negligible sampling error not identify \(\eta\)?
What changes if \(\rho=1\)? Explain why the general nonidentification argument no longer concerns the population event probability.
Does observing that a particular completed-data model fits the observed response and outcome frequencies establish its value of \(\eta\)? Explain.

Solutions. (1) Any \(\eta\in[0,1]\) is compatible. The endpoints are \(\pi_{\min}=0.75(0.20)=0.15\) and \(\pi_{\max}=0.15+0.25=0.40\), so \(\pi\in[0.15,0.40]\). (2) At \(\eta=0.40\), \(\pi=0.50(0.40)+0.50(0.40)=0.40\); at \(\eta=0.80\), \(\pi=0.20+0.50(0.80)=0.60\). Both retain the same observed probabilities \(P(R=1,Y=1)=0.20\) and \(P(R=1,Y=0)=0.30\). (3) Exact knowledge of \(\rho\) and \(\theta\) still supplies no observed outcomes for \(R=0\); every \(\eta\) in \([0,1]\) gives the same observed distribution. (4) If \(\rho=1\), all outcomes are observed and \(\pi=\theta\); there is no nonrespondent group whose outcome distribution could vary. (5) No. Fit to the observed frequencies checks only the observable part of the distribution. The chosen value of \(\eta\) concerns unobserved outcomes and needs a restriction or information beyond those frequencies.