Missingness Patterns, Indicators, and Observed Data

Representing an Incomplete Record

A multivariate record may contain some measured coordinates and some unobserved ones. To write a likelihood for such a record, we need to specify which coordinates were observed as well as their values. A vector with blanks alone is an informal description; response indicators and missingness patterns turn it into a mathematical object.

Let $X=(X_1,\ldots,X_p)^\mathsf{T}$ be a random vector with joint density or mass function $f_\theta(x)$, where $\theta$ indexes its distribution. For each coordinate, define $R_j=1$ if $X_j$ is observed and $R_j=0$ if it is not. Write $R=(R_1,\ldots,R_p)^\mathsf{T}$. The observed coordinates are indexed by the set $A=\{j:R_j=1\}$, and the missing coordinates by its complement $A^c$. The corresponding subvectors are $X_A$ and $X_{A^c}$.

Definition: The response indicator $R_j$ records whether coordinate $j$ is observed. A missingness pattern is a particular value $r\in\{0,1\}^p$, or equivalently its observed-coordinate set $A=\{j:r_j=1\}$. For pattern $A$, the observed data from one case are the pair $(A,x_A)$: the pattern and the values of the coordinates it identifies.

The pattern is part of the observed record. It is not generally enough to retain only the numerical values that happened to be observed: the pattern tells us which coordinates those values belong to. Nor should a missing entry be replaced with a numerical placeholder and treated as if it were an actual measurement. For instance, a missing value is not the same observation as a measured zero.

When $X$ is continuous, a convenient encoding of the observed values is $R$ together with $R\odot X$, where $\odot$ denotes coordinate-wise multiplication. The indicators resolve the ambiguity when a genuine observed value is zero. Pattern notation is often cleaner: for each possible $A$, record $x_A$ only in the coordinates named by $A$.

The Observed-Data Density

Suppose the response mechanism is described by probabilities $g_\eta(r\mid x)=P_\eta(R=r\mid X=x)$. This notation allows the probability of a pattern to depend on the full vector, including coordinates that end up unobserved. The joint model for $(X,R)$ then has density, with respect to the full-data measure and counting measure on patterns,

p_{\theta,\eta}(x,r)=f_\theta(x)g_\eta(r\mid x).

For pattern $A$, the observer sees $x_A$, but not $x_{A^c}$. The density for what is seen is obtained by marginalizing the joint density over the unobserved coordinates.

Theorem: Assume $X$ has density $f_\theta$ with respect to a product measure $\nu=\nu_A\otimes\nu_{A^c}$, and $R$ takes values in the finite set $\{0,1\}^p$ with conditional probabilities $g_\eta(r\mid x)$. For a pattern $r$ with observed-coordinate set $A$, the joint density of the observed record $(R=r,X_A=x_A)$, with respect to counting measure on $r$ and $\nu_A$ on $x_A$, is $$ q_{\theta,\eta}(r,x_A) = \int f_\theta(x_A,x_{A^c})g_\eta(r\mid x_A,x_{A^c})\,d\nu_{A^c}(x_{A^c}). $$ The integral is a sum when the missing coordinates are discrete. The formula also covers $A=\varnothing$, where it gives the probability of the all-missing pattern, and $A=\{1,\ldots,p\}$, where there are no coordinates to integrate out.

Proof. Let $B$ be a measurable set of possible observed values $x_A$. The probability of observing pattern $r$ and an observed subvector in $B$ is, by the joint model,

\begin{aligned} P_{\theta,\eta}(R=r,\ X_A\in B) &=\int_{\{x:x_A\in B\}} f_\theta(x)g_\eta(r\mid x)\,d\nu(x)\\ &=\int_B\left\{\int f_\theta(x_A,x_{A^c})g_\eta(r\mid x_A,x_{A^c}) \,d\nu_{A^c}(x_{A^c})\right\}d\nu_A(x_A). \end{aligned}

The integrand is nonnegative, so Tonelli’s theorem permits this order of integration without an additional integrability assumption. The expression in braces is therefore a density for the observed subvector together with pattern $r$, as claimed. Summing its integral over all $r$ gives one because $\sum_r g_\eta(r\mid x)=1$ for each $x$. $\square$

The displayed formula is a joint density for the pattern and observed values. If the analysis instead conditions on a particular pattern, its density is $q_{\theta,\eta}(r,x_A)/P_{\theta,\eta}(R=r)$, provided $P_{\theta,\eta}(R=r)>0$. These are different likelihood contributions. Which one is appropriate depends on the sampling and modeling setup; the pattern cannot be silently discarded.

Worked Examples

Worked Example: An Exponential Pair With One Missing Coordinate

Let $X_1$ and $X_2$ be independent exponential random variables with rates $\lambda>0$ and $\mu>0$. Thus, for positive coordinates,

f_{\lambda,\mu}(x_1,x_2) =\lambda e^{-\lambda x_1}\mu e^{-\mu x_2}.

Suppose $X_1$ is always observed, while $X_2$ is observed with conditional probability $e^{-a x_2}$, where $a>0$. Hence the pattern with only $X_1$ observed has conditional probability $1-e^{-a x_2}$. Find the observed-data density for that pattern.

For the pattern $A=\{1\}$, the observed value is $x_1$ and $x_2$ must be integrated out. The required response probability is $1-e^{-a x_2}$, so

\begin{aligned} q_{\lambda,\mu}(r=(1,0),x_1) &=\int_0^\infty \lambda e^{-\lambda x_1}\mu e^{-\mu x_2} (1-e^{-a x_2})\,dx_2\\ &=\lambda e^{-\lambda x_1} \left\{\int_0^\infty \mu e^{-\mu x_2}\,dx_2 -\int_0^\infty \mu e^{-(\mu+a)x_2}\,dx_2\right\}\\ &=\lambda e^{-\lambda x_1} \left(1-\frac{\mu}{\mu+a}\right)\\ &=\lambda e^{-\lambda x_1}\frac{a}{\mu+a}, \qquad x_1>0. \end{aligned}

The marginal density of $X_1$ alone is $\lambda e^{-\lambda x_1}$, but that is not the joint density of this observed value and the pattern. The factor $a/(\mu+a)$ is the probability of the pattern under the stated model. As a check, the probability of observing $X_2$ is $\int_0^\infty \mu e^{-\mu x_2}e^{-a x_2}\,dx_2=\mu/(\mu+a)$. The two pattern probabilities sum to one.

Worked Example: Summing Over a Missing Binary Coordinate

Let $X_1,X_2$ be binary with joint probabilities $P(X_1=x_1,X_2=x_2)$ given below. Suppose the response pattern is fixed at $R=(1,0)$: $X_1$ is observed and $X_2$ is missing. Find the likelihood contribution for observing $X_1=1$.

$X_1$	$X_2$	Joint probability
0	0	0.10
0	1	0.20
1	0	0.30
1	1	0.40

Assume for this calculation that the stated pattern occurs with probability one, so its conditional response probability is $g(r\mid x)=1$ for every $x$. Since the unobserved coordinate can be either 0 or 1, sum the two compatible full-data probabilities:

$$ q(r=(1,0),x_1=1) =P(X_1=1,X_2=0)+P(X_1=1,X_2=1) =0.30+0.40=0.70. $$

Thus $0.70$ is the probability of the observed record under this model. The probability $0.40$ alone is not the contribution: it is the probability of one particular complete record, $(1,1)$, which is more specific than what was observed.

Worked Example: A Gaussian Record Observed in Two Coordinates

Suppose $X\sim N(\mu,\Sigma)$ in three dimensions, with $\mu=(0,1,-1)^\mathsf{T}$ and $\Sigma=\begin{pmatrix}4&1&0\\1&9&3\\0&3&4\end{pmatrix}$. Consider a record with pattern $A=\{1,3\}$ and observed values $x_1=2,\ x_3=1$. Assume the pattern has probability one. Find the observed-data density.

A multivariate normal vector’s subvector is normal with the corresponding subvector of the mean and principal submatrix of the covariance. Here

X_A= \begin{pmatrix}X_1\\X_3\end{pmatrix} \sim N\left[ \begin{pmatrix}0\\-1\end{pmatrix}, \begin{pmatrix}4&0\\0&4\end{pmatrix} \right].

The observed deviations are $(2,2)^\mathsf{T}$. The determinant of the displayed covariance matrix is $16$, and its inverse is $\frac14 I_2$. The bivariate normal density is therefore

\begin{aligned} q(r,x_A) &=\frac{1}{(2\pi)^{2/2}|4I_2|^{1/2}} \exp\left\{-\frac12 \begin{pmatrix}2&2\end{pmatrix} \frac14 I_2 \begin{pmatrix}2\\2\end{pmatrix}\right\}\\ &=\frac{1}{8\pi}\exp\left\{-\frac12(1+1)\right\} =\frac{e^{-1}}{8\pi}. \end{aligned}

The value $X_2$ is not set to zero or otherwise imputed. Its possible values have already been integrated out in taking the marginal distribution of $(X_1,X_3)$.

Likelihoods Across Independent Records

For $n$ independent cases generated from the same joint model, write the observed record for case $i$ as $(r_i,x_{i,A_i})$, where $A_i=\{j:r_{ij}=1\}$. Independence gives the observed-data likelihood

L(\theta,\eta) =\prod_{i=1}^n \int f_\theta(x_{i,A_i},x_{i,A_i^c}) g_\eta(r_i\mid x_{i,A_i},x_{i,A_i^c}) \,d\nu_{A_i^c}(x_{i,A_i^c}).

Each factor integrates over that case’s missing coordinates, which can differ from case to case. The likelihood is a product of observed-record densities, not a density evaluated at one artificially completed data matrix.

Result: If $n_A$ cases share pattern $A$, their contributions can be grouped as $$ \prod_{i=1}^n q_{\theta,\eta}(r_i,x_{i,A_i}) = \prod_{A\subseteq\{1,\ldots,p\}} \ \prod_{i:A_i=A}q_{\theta,\eta}(r_A,x_{i,A}), $$ where $r_A$ is the indicator vector for pattern $A$. This is only a regrouping of factors; it does not make records with the same pattern identical.

To verify the identity, each case $i$ belongs to exactly one set in the inner products, namely the one indexed by its observed-coordinate set $A_i$. Thus every original factor appears once, and no factor is added or removed. $\square$

Common Pitfall and Why the Pattern Matters

A frequent error is to replace the observed-data likelihood contribution by the marginal density of the observed coordinates, without checking the model for $R$. The marginal density is $\int f_\theta(x_A,x_{A^c})\,d\nu_{A^c}(x_{A^c})$. The joint density of the pattern and observed values instead includes $g_\eta(r\mid x)$ inside the integral. The two expressions agree under particular assumptions about the response mechanism, but not in general.

In the exponential example, the marginal density of $X_1$ was $\lambda e^{-\lambda x_1}$, while the joint density for observing $X_1=x_1$ and missing $X_2$ was $\lambda e^{-\lambda x_1}a/(\mu+a)$. Omitting the pattern probability would discard part of the specified joint model. Conversely, if the analysis conditions on the pattern, the conditional density requires dividing by the probability of that pattern. A likelihood calculation must therefore state whether it models the joint observed record or conditions on the pattern.

The general procedure is systematic: record the pattern, write the full-data density and response-pattern probability, and integrate or sum over exactly the coordinates that were not observed. This representation provides the starting point for later questions about assumptions on the response mechanism and particular models for incomplete multivariate data.

Check Your Understanding

For each question, identify what is observed and what must be marginalized.

For $p=4$, what observed-coordinate set corresponds to $r=(1,0,1,0)^\mathsf{T}$? Which coordinates are integrated out?
Why is the pair $(R,R\odot X)$ less ambiguous than $R\odot X$ alone when an observed coordinate can equal zero?
In the joint model $f_\theta(x)g_\eta(r\mid x)$, write the observed-data density for pattern $A=\{2\}$ when $p=3$.
When is the probability of an observed record conditional on its pattern obtained by dividing the joint observed-record density by a pattern probability?
Explain why the probability of one complete binary vector is not generally the likelihood contribution when only one of its coordinates was observed.