Observed-Data Likelihood for Gaussian Mean and Covariance

Why the Observed Coordinates Determine the Likelihood

A multivariate Gaussian model describes a complete vector, but an incomplete-data likelihood must describe what was actually recorded. If one person has variables 1 and 3 observed and another has only variable 2 observed, their likelihood contributions are not the same full-dimensional density. Each contribution is the density of that person’s observed subvector, using the corresponding marginal mean and covariance.

This distinction matters because marginalizing a Gaussian vector retains dependence among the coordinates that remain observed. The likelihood therefore uses the relevant principal submatrix of the covariance matrix—not a covariance matrix reconstructed from separate univariate or pairwise calculations.

Definition: Let $Y_i=(Y_{i1},\ldots,Y_{ip})^\mathsf{T}$, $i=1,\ldots,n$, be independent observations from $N_p(\mu,\Sigma)$, where $\mu\in\mathbb{R}^p$ and $\Sigma$ is positive definite. Let $O_i\subseteq\{1,\ldots,p\}$ be the set of coordinates observed for case $i$, and let $y_{i,O_i}$ be the corresponding observed values. Write $\mu_{O_i}$ for the subvector of $\mu$ and $\Sigma_{O_iO_i}$ for the principal submatrix of $\Sigma$ indexed by $O_i$.

The likelihood considered here is the likelihood for $(\mu,\Sigma)$ from the observed values, under an ignorable missingness mechanism. In the usual likelihood formulation, ignorability requires missing at random and distinct parameters for the data model and the missingness mechanism. Under those conditions, the mechanism’s factor can be omitted when making likelihood inference about $(\mu,\Sigma)$. If the mechanism is nonignorable, the observed-data likelihood generally also involves that mechanism.

The Marginal Gaussian Density for One Pattern

For a nonempty observed set $O$, the subvector $Y_O$ has a Gaussian distribution with mean $\mu_O$ and covariance $\Sigma_{OO}$. Its density at $y_O$, with $d=|O|$, is

f_O(y_O;\mu,\Sigma) = \frac{1}{(2\pi)^{d/2}|\Sigma_{OO}|^{1/2}} \exp\left\{-\frac{1}{2}(y_O-\mu_O)^\mathsf{T} \Sigma_{OO}^{-1}(y_O-\mu_O)\right\}.

The determinant and inverse here are those of the $d\times d$ observed-coordinate submatrix. The unobserved coordinates do not appear in this density. If $O$ is empty, the case contributes no information about $(\mu,\Sigma)$; its likelihood contribution is defined to be 1.

Theorem: Under the independent $N_p(\mu,\Sigma)$ model and an ignorable missingness mechanism, the observed-data likelihood for $(\mu,\Sigma)$, up to a factor not involving these parameters, is $$ L_{\mathrm{obs}}(\mu,\Sigma) = \prod_{i:|O_i|>0} f_{O_i}(y_{i,O_i};\mu,\Sigma). $$

To verify the marginal-density formula, let $A$ be the $d\times p$ coordinate-selection matrix that extracts the entries indexed by $O$. The characteristic function of $Y\sim N_p(\mu,\Sigma)$, evaluated at $A^\mathsf{T}t$, is

E\!\left[\exp\{it^\mathsf{T}AY\}\right] = \exp\left\{it^\mathsf{T}A\mu -\frac{1}{2}t^\mathsf{T}A\Sigma A^\mathsf{T}t\right\}.

This is the characteristic function of a $d$-variate Gaussian vector with mean $A\mu=\mu_O$ and covariance $A\Sigma A^\mathsf{T}=\Sigma_{OO}$. Since $\Sigma$ is positive definite, every nonempty principal submatrix $\Sigma_{OO}$ is positive definite, so the stated Gaussian density exists.

For the likelihood claim, first consider the observed values and pattern for one case. Write $g(r\mid y)$ for the probability of pattern $r$ given the complete vector $y$. Under missing at random, for a fixed pattern $r$, this probability depends on $y$ only through the coordinates observed under $r$. Thus it can be written $g(r\mid y_O)$. Integrating out the unobserved coordinates gives

\begin{aligned} p(y_O,r;\mu,\Sigma) &=\int f(y_O,y_{O^c};\mu,\Sigma)\, g(r\mid y_O)\,dy_{O^c}\\ &=g(r\mid y_O)\int f(y_O,y_{O^c};\mu,\Sigma)\,dy_{O^c}\\ &=g(r\mid y_O)\,f_O(y_O;\mu,\Sigma). \end{aligned}

When the mechanism has distinct parameters, its factor does not involve $(\mu,\Sigma)$. For independent cases, multiplying the observed-data contributions therefore gives the theorem. If missingness is nonignorable, $g$ can depend on unobserved values; it cannot in general be taken outside the integral, and the likelihood need not reduce to a Gaussian marginal density. $\square$

Likelihood Across Arbitrary Missingness Patterns

The formula applies case by case, even when every case has a different observed set. If many cases share a pattern, it is convenient to group their contributions. Let $\mathcal{R}$ be the collection of distinct nonempty observed sets in the sample, and let $I_O=\{i:O_i=O\}$ and $n_O=|I_O|$. Taking logarithms gives

\ell_{\mathrm{obs}}(\mu,\Sigma) = -\frac{1}{2}\sum_{O\in\mathcal{R}} \left[ n_O|O|\log(2\pi) +n_O\log|\Sigma_{OO}| +\sum_{i\in I_O} (y_{i,O}-\mu_O)^\mathsf{T} \Sigma_{OO}^{-1}(y_{i,O}-\mu_O) \right].

Every pattern has its own dimension, determinant, and quadratic forms. The same $\mu$ and $\Sigma$ govern all patterns, but each contribution uses only the coordinates observed for that case. This grouped expression is an algebraic reorganization of the product likelihood; it does not assume that different patterns have different Gaussian parameters.

Worked Example: A Two-Coordinate Marginal

Suppose $Y\sim N_3(\mu,\Sigma)$, with \[ \mu=(0,1,2)^\mathsf{T},\qquad \Sigma= \begin{pmatrix} 4&1&0.5\\ 1&9&2\\ 0.5&2&1 \end{pmatrix}. \] The covariance is positive definite: its leading principal minors are $4$, $35$, and $18.75$, all positive. A case has coordinates 1 and 3 observed, with values $y_O=(1,3)^\mathsf{T}$.

The observed mean and covariance are \[ \mu_O=(0,2)^\mathsf{T},\qquad \Sigma_{OO}= \begin{pmatrix}4&0.5\\0.5&1\end{pmatrix}. \] The determinant is $4(1)-0.5^2=3.75$, and the inverse is \[ \Sigma_{OO}^{-1} =\frac{1}{3.75} \begin{pmatrix}1&-0.5\\-0.5&4\end{pmatrix}. \] The residual is $y_O-\mu_O=(1,1)^\mathsf{T}$, so the quadratic form is \[ (1,1)\Sigma_{OO}^{-1}(1,1)^\mathsf{T} =\frac{1-1+4}{3.75} =\frac{16}{15}. \] Therefore this case contributes \[ f_O(y_O;\mu,\Sigma) =\frac{1}{2\pi\sqrt{3.75}}\exp\left(-\frac{8}{15}\right). \]

The covariance between coordinates 1 and 3 is $0.5$, so it remains in the marginal covariance. The covariance involving coordinate 2 does not enter this particular contribution.

Worked Example: Multiplying Contributions From Different Patterns

Let $Y_i\sim N_2(\mu,\Sigma)$ independently, with \[ \mu=(0,1)^\mathsf{T},\qquad \Sigma=\begin{pmatrix}1&0.5\\0.5&4\end{pmatrix}. \] Three cases are observed: $y_1=(1,2)^\mathsf{T}$, $y_2=2$ on coordinate 1 only, and $y_3=1.5$ on coordinate 2 only. The matrix is positive definite because its determinant is $4-0.25=3.75>0$ and its first diagonal entry is positive.

For case 1, the residual is $(1,1)^\mathsf{T}$. The inverse covariance is \[ \Sigma^{-1}=\frac{1}{3.75} \begin{pmatrix}4&-0.5\\-0.5&1\end{pmatrix}, \] so the quadratic form is $(4-1+1)/3.75=16/15$. Its contribution is \[ f_{\{1,2\}}(y_1)=\frac{1}{2\pi\sqrt{3.75}}\exp(-8/15). \]

For case 2, the marginal distribution of coordinate 1 is $N(0,1)$, giving contribution \[ f_{\{1\}}(2)=\frac{1}{\sqrt{2\pi}}\exp(-2). \] For case 3, coordinate 2 has marginal distribution $N(1,4)$. Its residual is $0.5$, and its standardized quadratic term is $0.5^2/4=1/16$. Thus \[ f_{\{2\}}(1.5)=\frac{1}{2\sqrt{2\pi}}\exp(-1/32). \]

The observed-data likelihood at these parameter values is the product of the three contributions: \[ L_{\mathrm{obs}} =\frac{\exp(-8/15-2-1/32)} {(2\pi\sqrt{3.75})\sqrt{2\pi}\,2\sqrt{2\pi}}. \] Each case uses its own observed-coordinate marginal; the singleton cases do not contribute a bivariate density.

Worked Example: Grouping Repeated Patterns

Consider three independent observations from $N_2(\mu,\Sigma)$, with $\mu=(0,0)^\mathsf{T}$ and $\Sigma=\begin{pmatrix}1&0.4\\0.4&4\end{pmatrix}$. The observed data are $Y_{11}=0$, $Y_{21}=2$, and $Y_{32}=1$; each case has only the indicated coordinate observed. The covariance matrix is positive definite since $1(4)-0.4^2=3.84>0$.

The first two cases share pattern $O=\{1\}$, whose marginal variance is 1. Their contributions are $(2\pi)^{-1/2}$ and $(2\pi)^{-1/2}\exp(-2)$. The third case has pattern $\{2\}$, whose marginal variance is 4, and contributes $(2\sqrt{2\pi})^{-1}\exp(-1/8)$. Hence \[ L_{\mathrm{obs}} =\frac{\exp(-2-1/8)}{2(2\pi)^{3/2}}, \qquad \ell_{\mathrm{obs}} =-\frac{3}{2}\log(2\pi)-\log 2-\frac{17}{8}. \]

Although the full covariance has off-diagonal entries $0.4$, none appears in these three univariate marginal densities. There is no case with both coordinates observed to contribute a joint density involving that covariance.

Common Pitfall and Practical Interpretation

A common mistake is to multiply the full $p$-variate Gaussian density for every case, filling in missing coordinates with zero, a mean, or another convenient value. That calculation treats invented values as observations and yields a different likelihood. Another tempting shortcut is to multiply univariate densities for each observed coordinate, even when several coordinates are jointly observed. That discards their covariance and is generally incorrect.

The correction is to form one observed subvector per case and use its marginal density. If coordinates $j$ and $k$ are jointly observed, their contribution uses the $2\times2$ submatrix with diagonal entries $\Sigma_{jj},\Sigma_{kk}$ and off-diagonal entries $\Sigma_{jk}$. If they are observed in separate cases, each case instead contributes its own univariate marginal density.

Key takeaway: For an ignorable missingness mechanism, each case contributes the Gaussian marginal density on exactly its observed coordinates. Multiply those contributions across cases, or group them by pattern; do not insert values for missing entries or discard dependence among coordinates observed together.

This construction supplies the likelihood target for inference about a Gaussian mean and covariance under incomplete observation. The pattern-specific determinants and quadratic forms also make clear which parts of the covariance matrix enter each contribution. Whether the collected patterns provide enough information to estimate every parameter is a separate question; the likelihood formula itself does not guarantee that all parameters are identifiable.

Check Your Understanding

For each question, identify which marginal likelihood contribution is appropriate before doing any calculation.

If $Y\sim N_4(\mu,\Sigma)$ and only coordinates 2 and 4 are observed, which parts of $\mu$ and $\Sigma$ enter the contribution?
Why does the marginal density for a jointly observed pair generally require the covariance between those two coordinates?
State the conditions under which the missingness mechanism can be omitted from likelihood inference for $(\mu,\Sigma)$.
A case has no observed coordinates. What is its contribution to the likelihood for $(\mu,\Sigma)$, and why?
Two coordinates are each observed in the data, but never in the same case. Do their separate univariate contributions directly contain their covariance? Explain.

Observed-Data Likelihood for Gaussian Mean and Covariance

What You'll Learn