Gaussian Likelihood From Independent Observations

From Independent Observations to a Likelihood

In a multivariate normal model, each observation is a vector, not a collection of unrelated scalar measurements. Its covariance matrix describes both the variability of each coordinate and the dependence between coordinates. When several such vectors are observed independently, their joint density is the starting point for likelihood-based inference.

The central task here is to form that joint likelihood carefully. In particular, a factor may be omitted when comparing parameter values only if it does not depend on any unknown parameter. A term that looks like a constant with respect to the data may still vary with the mean or covariance, and therefore may not be discarded.

Definition: Let $X_1,\ldots,X_n$ be independent random vectors in $\mathbb{R}^p$, each with distribution $N_p(\mu,\Sigma)$, where $\mu\in\mathbb{R}^p$ and $\Sigma$ is a symmetric positive-definite $p\times p$ matrix. For observed vectors $x_1,\ldots,x_n$, the likelihood is their joint density, viewed as a function of the unknown parameter $(\mu,\Sigma)$: $$ L(\mu,\Sigma;x_1,\ldots,x_n) =\prod_{i=1}^n f_{\mu,\Sigma}(x_i). $$

The parameter $\mu$ is the common location vector. The covariance parameter $\Sigma$ contains $p(p+1)/2$ distinct unknown entries: its $p$ variances on the diagonal and its $p(p-1)/2$ covariances above the diagonal. Positive definiteness is required for the ordinary nonsingular multivariate normal density. Within each observation, coordinates need not be independent; their dependence is represented by the off-diagonal entries of $\Sigma$. Independence in the model is between the vectors $X_i$.

The Single-Observation Density and the Joint Formula

For a positive-definite $\Sigma$, the density of one $p$-dimensional observation $x$, with respect to $p$-dimensional Lebesgue measure, is

f_{\mu,\Sigma}(x) =(2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\!\left\{-\frac12(x-\mu)^{\mathsf T}\Sigma^{-1}(x-\mu)\right\}.

Here $|\Sigma|$ is the determinant. The quadratic form measures the squared deviation from $\mu$ using the covariance geometry. The exponent is nonpositive because $\Sigma^{-1}$ is positive definite.

Theorem: If $X_1,\ldots,X_n$ are independent and each has distribution $N_p(\mu,\Sigma)$, then the joint likelihood for observed $x_1,\ldots,x_n$ is $$ L(\mu,\Sigma;x_1,\ldots,x_n) =(2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\!\left\{-\frac12\sum_{i=1}^n (x_i-\mu)^{\mathsf T}\Sigma^{-1}(x_i-\mu)\right\}. $$

Proof. Independence means that the joint density of the random vectors is the product of their individual densities. Substituting the single-observation density gives

\begin{aligned} \prod_{i=1}^n f_{\mu,\Sigma}(x_i) &=\prod_{i=1}^n \left[(2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\!\left\{-\frac12(x_i-\mu)^{\mathsf T} \Sigma^{-1}(x_i-\mu)\right\}\right]\\ &=(2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\!\left\{-\frac12\sum_{i=1}^n (x_i-\mu)^{\mathsf T}\Sigma^{-1}(x_i-\mu)\right\}. \end{aligned}

The second equality combines the $n$ identical normalizing factors, uses $(|\Sigma|^{-1/2})^n=|\Sigma|^{-n/2}$, and uses the product rule for exponentials. This product is the joint density evaluated at the observed sample, regarded as a function of $(\mu,\Sigma)$. $\square$

Which Factors Depend on the Unknowns?

The likelihood is a function of the parameters once the observed data have been fixed. This viewpoint is essential for classifying its terms. The factor $(2\pi)^{-np/2}$ depends on the sample size and dimension, but not on $\mu$ or $\Sigma$. By contrast, $|\Sigma|^{-n/2}$ depends on the unknown covariance whenever $\Sigma$ is unknown. The exponential term also depends on the unknowns through both $\mu$ and $\Sigma^{-1}$.

If only $\mu$ is unknown and $\Sigma$ is fixed, then the determinant factor is constant with respect to the unknown parameter, as well as the $2\pi$ factor. If $\Sigma$ is unknown, the determinant factor cannot be discarded when comparing candidate covariance matrices. A useful test is to write down the parameter space first and ask whether a factor changes as a parameter in that space changes.

Result: Suppose two likelihood expressions $L_1(\theta)$ and $L_2(\theta)$ satisfy $L_1(\theta)=cL_2(\theta)$ throughout the parameter space, where $c>0$ does not depend on $\theta$. They have the same likelihood ratios and the same maximizers, if maximizers exist.

Proof. For any two parameter values $\theta_a,\theta_b$,

\frac{L_1(\theta_a)}{L_1(\theta_b)} =\frac{cL_2(\theta_a)}{cL_2(\theta_b)} =\frac{L_2(\theta_a)}{L_2(\theta_b)}.

Thus all pairwise likelihood comparisons are unchanged. Also, since $c>0$, $L_1(\theta_a)\geq L_1(\theta_b)$ if and only if $L_2(\theta_a)\geq L_2(\theta_b)$. Consequently the two functions have the same set of maximizers whenever that set is attained. This justifies writing likelihoods up to a positive parameter-free factor for such comparisons; it does not make the omitted factor part of the exact density disappear. $\square$

Worked Example: Two Scalar Normal Observations

Suppose $n=2$, $p=1$, and the observations are $x_1=1$ and $x_2=3$. In one dimension, $\mu$ is a scalar and $\Sigma=\sigma^2$, with $\sigma^2>0$. The likelihood is

\begin{aligned} L(\mu,\sigma^2) &=(2\pi\sigma^2)^{-1} \exp\!\left\{-\frac{(1-\mu)^2+(3-\mu)^2}{2\sigma^2}\right\}. \end{aligned}

For example, at $\mu=2$ and $\sigma^2=1$, the squared deviations are $(1-2)^2=1$ and $(3-2)^2=1$. Their sum is $2$, so

L(2,1)=(2\pi)^{-1}e^{-2/2}=\frac{e^{-1}}{2\pi}.

Only the factor $(2\pi)^{-1}$ is parameter-free when both $\mu$ and $\sigma^2$ are unknown. The factor $(\sigma^2)^{-1}$ and the exponential both vary with the parameters, so omitting either would change likelihood comparisons.

Worked Example: A Bivariate Sample With Correlated Coordinates

Take two observed vectors $x_1=(1,0)^{\mathsf T}$ and $x_2=(0,1)^{\mathsf T}$. Evaluate their joint density at $\mu=(0,0)^{\mathsf T}$ and

\Sigma=\begin{pmatrix}4&1\\1&1\end{pmatrix}.

The determinant is $4(1)-1(1)=3>0$, and the leading diagonal entry is positive, so this symmetric matrix is positive definite. Its inverse is

\Sigma^{-1} =\frac13\begin{pmatrix}1&-1\\-1&4\end{pmatrix}.

For $x_1$, the quadratic form is

x_1^{\mathsf T}\Sigma^{-1}x_1 =(1,0)\frac13\begin{pmatrix}1\\-1\end{pmatrix} =\frac13.

For $x_2$, it is

x_2^{\mathsf T}\Sigma^{-1}x_2 =(0,1)\frac13\begin{pmatrix}-1\\4\end{pmatrix} =\frac43.

The sum is $5/3$. Since $n=p=2$, the normalizing factor is $(2\pi)^{-2}|\Sigma|^{-1}=(4\pi^2)^{-1}(1/3)$. Therefore the joint density at these parameter values is

L((0,0)^{\mathsf T},\Sigma) =\frac{1}{12\pi^2}\exp\!\left(-\frac12\cdot\frac53\right) =\frac{e^{-5/6}}{12\pi^2}.

The nonzero off-diagonal entries of $\Sigma$ affect its inverse and hence both quadratic forms. The coordinates of each observed vector have not been treated as independent.

Worked Example: A Covariance Matrix Known in Advance

Let $n=p=2$, with observations $x_1=(1,2)^{\mathsf T}$ and $x_2=(2,0)^{\mathsf T}$. Suppose the covariance is known to be $\Sigma=\begin{pmatrix}1&0\\0&4\end{pmatrix}$, while $\mu$ is unknown. The determinant is $4$, and the inverse is $\operatorname{diag}(1,1/4)$. At $\mu_0=(0,0)^{\mathsf T}$, the quadratic forms are

1^2+\frac{2^2}{4}=2, \qquad 2^2+\frac{0^2}{4}=4.

Their sum is $6$, and the likelihood at $\mu_0$ is

L(\mu_0,\Sigma) =(2\pi)^{-2}4^{-1}e^{-6/2} =\frac{e^{-3}}{16\pi^2}.

At $\mu_1=(1,1)^{\mathsf T}$, the residuals are $(0,1)^{\mathsf T}$ and $(1,-1)^{\mathsf T}$. Their quadratic forms are $0^2+1^2/4=1/4$ and $1^2+(-1)^2/4=5/4$, whose sum is $3/2$. Thus

L(\mu_1,\Sigma) =\frac{1}{16\pi^2}e^{-(3/2)/2} =\frac{e^{-3/4}}{16\pi^2}, \qquad \frac{L(\mu_1,\Sigma)}{L(\mu_0,\Sigma)} =e^{9/4}.

Here both the $2\pi$ and determinant factors are constant with respect to the unknown $\mu$, so they cancel in the ratio. The quadratic-form contribution does not cancel because it changes with $\mu$.

Common Pitfall: Calling a Parameter-Dependent Factor a Constant

A frequent error is to call $|\Sigma|^{-n/2}$ a constant merely because it does not involve the observed vectors $x_i$. Likelihood calculations hold the data fixed and vary the unknown parameters. If $\Sigma$ is unknown, changing $\Sigma$ generally changes its determinant, so this factor is not constant for the likelihood problem.

The correction is to specify what is held fixed. A factor may depend on the data, the sample size, and the dimension while remaining constant in the parameters; it can then be omitted for likelihood comparisons. Conversely, a factor that does not involve the data can still depend on an unknown parameter and must be retained. The same likelihood expression can therefore have different parameter-free factors under different models—for example, when a covariance is unknown versus fixed by assumption.

The distinction matters because the likelihood balances two features of the covariance: its determinant controls the density's scale, while the quadratic forms measure the observations' deviations using the inverse covariance. Keeping only the quadratic forms when covariance is unknown discards part of the model's dependence on that parameter.

Check Your Understanding

Use the definitions and likelihood formula to answer each question.

For $n$ independent $N_p(\mu,\Sigma)$ observations, which factors in the joint likelihood depend on $\Sigma$ when it is unknown?
Why does independence of the observation vectors justify multiplying their densities, even when coordinates within a vector are correlated?
If $\Sigma$ is known and only $\mu$ is unknown, which factors in the displayed joint likelihood can be omitted for comparing values of $\mu$?
Explain why a factor that depends on the data but not on the unknown parameters can be omitted in likelihood comparisons.
For a single observation, what role does positive definiteness of $\Sigma$ play in the multivariate normal density?

Gaussian Likelihood From Independent Observations

What You'll Learn