Observable Implications and Untestable Restrictions

What Can the Observed Data Constrain?

A restriction on a missing-data model may concern quantities that are recorded, quantities that are never recorded, or both. The key question is not simply whether a restriction is plausible. It is whether imposing it changes which distributions of the observed data the model can produce. If it does, the restriction has observable implications. If it changes only the possible values assigned to unobserved outcomes while leaving the set of observable distributions unchanged, the data cannot test that restriction.

This distinction is central to interpreting model checks. An observed-data test can reveal a conflict between the data and a model’s observable implications. But a model may also impose assumptions about missing outcomes that no observed-data test can assess. A model can therefore satisfy every testable implication and still rely on restrictions about unobserved outcomes that the data do not determine.

Definition: Let \(P\) be a full-data distribution and let \(g(P)\) be the distribution of the variables actually observed under the response pattern. For a model collection \(\mathcal{M}\), its observable distributions are \(\mathcal{O}(\mathcal{M})=\{g(P):P\in\mathcal{M}\}\). Relative to a reference model \(\mathcal{M}_0\), a restriction defining a smaller model \(\mathcal{M}_1\subseteq\mathcal{M}_0\) is testable from the observed data if \(\mathcal{O}(\mathcal{M}_1)\) is a proper subset of \(\mathcal{O}(\mathcal{M}_0)\). It is untestable relative to that reference model if the two sets of observable distributions are equal, even though the restriction rules out some full-data distributions.

The comparison is relative to a reference model: a restriction is not “testable” in isolation from the other assumptions that define the model. Several restrictions can also have implications jointly even if one of them, considered alone, does not. In practice, begin with a clear description of what is observed, then ask which probability statements can be calculated from that observed distribution.

A Binary Outcome: Different Completions, Same Observations

Let \(Y\) be a binary outcome and let \(R=1\) mean that \(Y\) is observed, while \(R=0\) means that it is missing. The observed data reveal the probabilities of \((R=1,Y=1)\), \((R=1,Y=0)\), and \(R=0\). They do not reveal the distribution of \(Y\) among those with \(R=0\).

Write \(a=P(R=1,Y=1)\), \(b=P(R=1,Y=0)\), and \(m=P(R=0)\), so \(a+b+m=1\). When \(m>0\), define \(\theta=P(Y=1\mid R=0)\). The observed-data law fixes \(a\), \(b\), and \(m\), but it does not fix \(\theta\). The overall prevalence of a positive outcome is

P(Y=1)=a+m\theta.

This equation gives a short justification for the nonidentification: changing \(\theta\) changes the complete-data prevalence, but leaves all three observed probabilities \(a\), \(b\), and \(m\) unchanged. Thus, any restriction on \(\theta\) alone can rule out some full-data laws without necessarily ruling out any observed-data law.

Worked Example: Two Prevalences Compatible With the Same Observations

Hypothetical data: In a population of 100 people, suppose 60 have an observed outcome: 30 are positive and 30 are negative. The remaining 40 outcomes are missing. The observed proportions are \(a=0.30\), \(b=0.30\), and \(m=0.40\).

Consider first a completion with \(\theta=0.10\). The implied overall positive proportion is

$$ P(Y=1)=0.30+(0.40)(0.10)=0.34. $$

In a hypothetical completed group of 100, this corresponds to 30 observed positives plus 4 positives among the 40 people with missing outcomes: \(34/100=0.34\).

Now consider \(\theta=0.90\). The implied overall positive proportion is

$$ P(Y=1)=0.30+(0.40)(0.90)=0.66. $$

This completion has 30 observed positives plus 36 positives among the people with missing outcomes: \(66/100=0.66\). Both completions have the same observed proportions, namely 30% observed positive, 30% observed negative, and 40% missing. The observed data do not distinguish the two overall prevalences.

The assumptions behind this calculation are that \(Y\) is binary, \(R\) correctly indicates whether it was recorded, and the stated proportions describe the population distribution. No missing-at-random assumption is needed to calculate the observed proportions. The choice of \(\theta\), however, is not determined by those proportions.

Restrictions That Do Imply Observable Constraints

A restriction is testable when it connects probabilities that the observed data reveal. For example, suppose \(Y_1\) is recorded for two response patterns: people with both \(Y_1\) and a later outcome \(Y_2\) observed, and people with \(Y_1\) observed but \(Y_2\) missing. A pattern-mixture restriction that the distribution of \(Y_1\) is the same in those two patterns is observable: \(Y_1\) is available in both groups. In a population, the restriction requires the corresponding conditional distributions to be equal.

P(Y_1=y\mid R=(1,1))=P(Y_1=y\mid R=(1,0))

Here \(R=(1,1)\) means both outcomes are observed, and \(R=(1,0)\) means \(Y_1\) is observed but \(Y_2\) is not. This restriction can be contradicted by the observed distribution of \(Y_1\) across the two patterns. By contrast, the same comparison for \(Y_2\) cannot be made between these patterns because \(Y_2\) is not recorded when \(R=(1,0)\).

Worked Example: Testing Equality of an Observed Outcome Across Patterns

Hypothetical data: Among 60 people with both outcomes observed, 36 have \(Y_1=1\). Among 40 people with \(Y_1\) observed and \(Y_2\) missing, 16 have \(Y_1=1\). Suppose the model restriction is equality of the population proportions of \(Y_1=1\) in these two patterns.

The observed proportions are \(36/60=0.60\) and \(16/40=0.40\). To illustrate a test of the equality restriction, use a two-sided pooled two-proportion \(z\)-test. This calculation assumes independent observations across people and uses the usual large-sample normal approximation. The pooled proportion under \(H_0\) is

\hat{p}=\frac{36+16}{60+40}=0.52.

The standard error under \(H_0\) is

SE=\sqrt{0.52(1-0.52)\left(\frac{1}{60}+\frac{1}{40}\right)} =\sqrt{0.2496(0.04167)} \approx 0.1020.

The test statistic is

z=\frac{0.60-0.40}{0.1020}\approx 1.961.

The two-sided \(p\)-value is approximately \(0.0499\). At the 5% significance level, this sample provides evidence against equality of the two population proportions. The test targets an observable implication because both groups’ \(Y_1\) values are recorded. The result does not establish that a particular alternative model is correct, and the \(p\)-value is not the probability that the equality restriction is true.

Restrictions About Outcomes That Are Missing

A pattern-mixture model describes outcome distributions conditional on response pattern. It can include restrictions on distributions that are observed within each pattern and restrictions on outcomes missing within a pattern. The first kind may imply observable constraints; the second kind, by itself, generally cannot be checked against outcomes that were never recorded. The distinction is about what information the response pattern makes available, not about whether a restriction is written as a mean, a probability, or a regression model.

Worked Example: A Restriction on a Missing Later Outcome

Continue with the hypothetical two-pattern setting. Suppose 60 people have both \(Y_1\) and \(Y_2\) observed, and 40 have \(Y_1\) observed but \(Y_2\) missing. Among the 60 complete cases, 30 have \(Y_2=1\), so \(P(Y_2=1\mid R=(1,1))\) is estimated by \(30/60=0.50\). Consider the restriction that the missing-pattern group would also have a 50% positive rate for \(Y_2\).

The observed data do not record \(Y_2\) for those 40 people. A completion assigning a 25% positive rate to that group would imply 10 positive missing outcomes and an overall positive rate of

\frac{30+40(0.25)}{100}=\frac{40}{100}=0.40.

A different completion assigning a 75% positive rate would imply 30 positive missing outcomes and an overall positive rate of

\frac{30+40(0.75)}{100}=\frac{60}{100}=0.60.

Both completions leave unchanged every recorded \(Y_1\), every \(Y_2\) among the 60 complete cases, and the response-pattern counts. The restriction that the two patterns have the same \(Y_2\) positive rate is therefore not testable using these data alone. It selects among compatible full-data distributions; it does not change the observed-data distribution.

This conclusion assumes that the only information about \(Y_2\) for the \(R=(1,0)\) group is its missingness indicator and the recorded variables described here. If additional observed measurements or external information are available, the question may change; whether they help depends on their relationship to the missing outcomes and the assumptions connecting them.

A Practical Audit and Common Pitfall

To assess a proposed restriction, first write down the observed variables for each response pattern. Then express the restriction as a probability statement and check whether every variable in that statement is observed in the group to which it refers. If it is, the statement may yield a testable implication. If it includes an unobserved outcome, ask whether other assumptions link that outcome to observed quantities strongly enough to induce a constraint on the observed distribution. A restriction involving a missing value is not automatically untestable, but its implications must be derived rather than assumed.

When a restriction does have observable implications, sampling variability matters. A sample will almost never satisfy a population equality exactly. Use an appropriate statistical test or interval to assess the discrepancy, and report the uncertainty. Failure to reject an implication is not proof that the restriction is true. Conversely, if the restriction has no observable implication under the stated model, a large sample of the same observed variables cannot test it merely by increasing precision.

Common mistake: “The fitted model reproduces the observed response rates, so its assumptions about the missing outcomes are supported.” Matching observed response rates does not check a restriction that only determines outcomes never recorded.
Correction: Separate the model’s observable implications from its assumptions about missing outcomes. Assess the former with observed data; describe the latter as assumptions unless additional identifying information is available.
Another mistake: “The restriction mentions a missing outcome, so none of the model can be tested.” A model may combine an untestable restriction about missing values with other restrictions that do constrain observed distributions.

The earlier material on observational equivalence and identification provides the broader perspective: multiple full-data laws can map to the same observed-data law. A restriction may reduce that set of compatible full-data laws without reducing the observable distributions. This is why model fit and identification are separate questions, and why assumptions about the unobserved part of the data must be reported distinctly from constraints that the observed data can assess.

Key takeaway: A restriction is testable only when it rules out some observable distributions relative to a stated reference model. Restrictions that merely choose among full-data distributions with the same observed-data law cannot be validated from those observed data alone.

Check Your Understanding

For each question, identify whether the restriction is testable from the stated observed data, and explain why. Solutions follow.

A binary outcome is observed for respondents and missing for nonrespondents. Can the observed data alone test the restriction that the nonrespondents’ positive-outcome rate is 0.30?
In two response patterns, \(Y_1\) is observed in both. Can the data test the restriction that the population mean of \(Y_1\) is equal across patterns?
Why does a non-significant test of equal observed \(Y_1\) means not prove that the population means are equal?
A model imposes both equality of observed response rates across two recorded covariate groups and a specified distribution for a missing outcome. Which restriction can be checked from the observed data?

Solutions. (1) No, not from those data alone: the nonrespondents’ outcomes are unobserved, and different positive-outcome rates can leave the observed law unchanged. (2) Yes, provided the relevant observations and pattern labels are recorded; both means are calculated from observed \(Y_1\) values, so equality is an observable population restriction. (3) A non-significant result means the sample did not provide sufficient evidence against equality at the chosen level; sampling variability and limited precision remain possible, so it does not establish equality. (4) Equality of response rates across recorded covariate groups is testable because both the covariates and response indicators are observed. The specified missing-outcome distribution is not testable from these data alone if it only concerns outcomes that were never recorded.