Nonignorable Missingness and Observational Equivalence

Why Missingness Can Hide Distinct Full-Data Distributions

When data are missing, the observed records do not directly reveal the values that were not recorded. If the probability of a value being missing depends on that value itself, the observed cases may give a misleading picture of the full population. More fundamentally, distinct population distributions and missingness processes can produce exactly the same observed records. In that case, collecting more records under the same process does not distinguish the explanations.

The issue is one of identification, not merely of precision. Identification asks whether the observed-data distribution uniquely determines a target feature of the full-data distribution. We will use a binary outcome to show the issue transparently, then extend the reasoning to an observed covariate.

Definitions: Missingness and Observational Equivalence

For a vector \(Y\), the response indicator is a vector \(R\) marking which components are observed, and the record includes the pattern \(R\) and the observed components \(Y_{\mathrm{obs}}\). For each possible pattern, nonignorability means that the probability of that pattern can vary with values of components unobserved under that pattern, after conditioning on the observed components and any fully observed covariates. In the binary examples below there are no covariates, so nonignorability simply means that \(P(R=1\mid Y=1)\) and \(P(R=1\mid Y=0)\) differ.

Write \(a=P(Y=1,R=1)\), \(b=P(Y=0,R=1)\), and \(c=P(R=0)\) for a binary outcome. These probabilities describe everything available in the observed records: the observed positive cases, observed negative cases, and missing cases. They satisfy \(a+b+c=1\). In particular, the records with \(R=0\) do not disclose whether their outcome was 0 or 1.

A Continuum of Compatible Full-Data Distributions

The two models above are not an isolated coincidence. For any observed cells with \(a>0\), \(b>0\), and \(c>0\), let \(u=P(Y=1,R=0)\), the probability mass of missing cases whose unobserved outcome is 1. Then \(P(Y=0,R=0)=c-u\). The observed data do not tell us \(u\), but every choice \(0\leq u\leq c\) assigns the same observed cells. For \(0<u<c\), the implied full-data prevalence and response probabilities are

These expressions follow by adding the observed and missing mass for each outcome, then dividing each observed cell by its full-data outcome probability. Because \(a,b>0\), the denominators are positive throughout this interval. Whenever the two response probabilities differ, this compatible model is nonignorable.

Proof. For any \(u\in[0,c]\), assign probabilities \(a+u\) to \(Y=1\) and \(b+c-u\) to \(Y=0\). Assign response probabilities \(a/(a+u)\) and \(b/(b+c-u)\), respectively, whenever the denominators are positive. The resulting observed joint probabilities are

The remaining probability is \(u+(c-u)=c\), all assigned to \(R=0\). Thus each allocation reproduces the same observed-data law, and the prevalence \(a+u\) ranges from \(a\) to \(a+c\). If a denominator is zero at an endpoint, the corresponding outcome has probability zero, so its response probability can be assigned arbitrarily without changing the law. This proves the result. \(\square\)

The interval is an identification bound, not a confidence interval: it describes values compatible with the population observed-data distribution itself. Sampling uncertainty would add a separate layer. Even if \(a\), \(b\), and \(c\) were known exactly, this argument would not select one value within the interval without additional information or assumptions.

Why More Observations Cannot Resolve Equivalence

The examples demonstrate failure of this definition for the full-data prevalence: the same observed law is compatible with different prevalences. This also gives a precise limit on what an estimator based only on observed records can achieve.

Proof. Since the observed-record distributions agree, the joint distribution of any sample of size \(n\) agrees under the two models. Therefore any estimator \(\hat{\theta}_n\) computed from that sample has the same distribution under both. Let \(\delta=|\theta_1-\theta_2|>0\) and choose \(\varepsilon=\delta/3\). The intervals \((\theta_1-\varepsilon,\theta_1+\varepsilon)\) and \((\theta_2-\varepsilon,\theta_2+\varepsilon)\) are disjoint. If the estimator were consistent under both models, its probability of falling in the first interval would tend to 1 under the first model, and its probability of falling in the second interval would tend to 1 under the second model. But the estimator has the same distribution under both, so these two probabilities are probabilities of disjoint events under one and the same distribution. Their sum cannot exceed 1, contradicting convergence of both to 1. Hence simultaneous consistency is impossible. \(\square\)

This is not a claim that every estimate from incomplete data is useless. It says that, without a restriction or source of information that distinguishes the equivalent models, the observed records alone cannot guarantee the correct full-data target. Increasing the sample size can estimate the observed-data probabilities more precisely, but it cannot separate models that agree on those probabilities exactly.

Interpretation and Common Pitfalls

The practical importance of observational equivalence is that conclusions about full-data quantities can depend on assumptions that are not testable from the observed records alone. In the binary calculation, the data determine \(a\), \(b\), and \(c\), but not \(u\), the allocation of missing cases across outcome values. A proposed single prevalence therefore reflects some additional restriction, external information, or modeling choice beyond the observed-data distribution.

• Mistake: “With a large enough sample, the true prevalence will become clear.” A larger sample can estimate the observed probabilities accurately. If two models give those probabilities exactly, the sample size does not distinguish them.
• Mistake: “The observed fraction of positive outcomes is the population prevalence.” The observed fraction targets \(P(Y=1\mid R=1)=a/(a+b)\), whereas the full-data prevalence is \(a+u\). These quantities need not agree.
• Mistake: “A fully observed covariate always fixes the problem.” The stratified example shows that distinct allocations of missing outcomes can remain compatible within every covariate stratum.

Additional information may narrow or remove the ambiguity—for example, a defensible restriction on the missingness process or outcome information obtained through a separate measurement effort. The essential check is whether that added information rules out one of the otherwise observationally equivalent models. A precise answer under a chosen assumption is not the same as identification from observed data alone.