Sufficiency and Minimal Sufficiency in Gaussian Models

Why Gaussian Data Can Be Summarized

In a multivariate Gaussian model, each observation is a vector, and the unknown covariance matrix describes how its coordinates vary and co-vary. A useful summary must therefore preserve information about both location and scatter. The likelihood gives a precise way to identify such a summary: if it can be factored into a term involving the data only through a statistic and a term involving the parameter, that statistic is sufficient.

Here \(X_1,\ldots,X_n\) are independent and identically distributed \(p\)-dimensional column vectors with \(X_i\sim N_p(\mu,\Sigma)\), where \(\mu\in\mathbb R^p\) is unrestricted and \(\Sigma\) is any positive-definite \(p\times p\) matrix. Write \(x=(x_1,\ldots,x_n)\) for an observed sample. The parameter space is the same for every sample, and the joint density is positive on all of \((\mathbb R^p)^n\).

Definition: A statistic \(T(X_1,\ldots,X_n)\) is sufficient for a parameter if, conditional on \(T\), the distribution of the full data does not depend on that parameter. In a dominated model, the factorization theorem says that \(T\) is sufficient if and only if the joint density can be written as \(g_\theta(T(x))h(x)\), with the parameter \(\theta\) appearing in \(g_\theta\) and not in \(h\).

The Gaussian Likelihood and a Sufficient Summary

For this model the joint density is

f_{\mu,\Sigma}(x) =(2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\!\left\{-\frac12\sum_{i=1}^n (x_i-\mu)'\Sigma^{-1}(x_i-\mu)\right\}.

Define the vector sum \(s(x)\) and the sum of outer products \(Q(x)\) by

s(x)=\sum_{i=1}^n x_i,\qquad Q(x)=\sum_{i=1}^n x_ix_i'.

Expanding each quadratic form and using \(\operatorname{tr}(ab')=b'a\) gives

\sum_{i=1}^n(x_i-\mu)'\Sigma^{-1}(x_i-\mu) = \operatorname{tr}\!\left(\Sigma^{-1}Q(x)\right) -2\mu'\Sigma^{-1}s(x) +n\mu'\Sigma^{-1}\mu.

Thus the likelihood depends on the observations only through \(s(x)\) and \(Q(x)\). The factorization has \(h(x)=1\), so the factorization theorem establishes the following result.

Theorem: In the Gaussian model with unknown mean and unrestricted positive-definite covariance, \(T(X)=(s(X),Q(X))\) is sufficient for \((\mu,\Sigma)\).

An equivalent summary is the sample mean and the centered scatter matrix. Define

\bar X=\frac{s(X)}{n},\qquad C=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'.

Expanding the centered products shows \(C=Q-n\bar X\bar X'\). Conversely, \(s=n\bar X\) and \(Q=C+n\bar X\bar X'\). These transformations determine each pair from the other, so \((\bar X,C)\) is also sufficient. One may use \(C/(n-1)\) instead when \(n>1\); dividing by a known nonzero constant does not change the information in the statistic.

Worked Example: Computing the Gaussian Summary

Suppose \(n=3\), \(p=2\), and the observed vectors are \(x_1=(1,0)'\), \(x_2=(0,1)'\), and \(x_3=(1,1)'\). Calculate both forms of the sufficient statistic.

Vector sum. Adding the observations gives \(s=(2,2)'\), so \(\bar x=(2/3,2/3)'\).

Sum of outer products. The three outer products are \(x_1x_1'=\begin{pmatrix}1&0\\0&0\end{pmatrix}\), \(x_2x_2'=\begin{pmatrix}0&0\\0&1\end{pmatrix}\), and \(x_3x_3'=\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Therefore \(Q=\begin{pmatrix}2&1\\1&2\end{pmatrix}\).

Centered scatter. Since \(C=Q-ss'/3\), and \(ss'/3=\begin{pmatrix}4/3&4/3\\4/3&4/3\end{pmatrix}\), we obtain \(C=\begin{pmatrix}2/3&-1/3\\-1/3&2/3\end{pmatrix}\). These quantities, rather than the ordering of the three observations, determine the likelihood for every \(\mu\) and \(\Sigma\).

Minimal Sufficiency and the Likelihood-Ratio Criterion

A sufficient statistic may contain more information than the likelihood needs. A minimal sufficient statistic is sufficient and, up to events of probability zero, is a function of every other sufficient statistic. For models with positive densities on a common support, the likelihood-ratio criterion characterizes minimal sufficiency: two sample points belong to the same minimal-sufficiency class exactly when their density ratio is constant as the parameter varies.

More precisely, for a dominated family with common support, take densities that are positive there. A statistic \(T\) is minimal sufficient if, for any two sample points \(x,y\) in the support,

T(x)=T(y) \quad\Longleftrightarrow\quad \frac{f_\theta(x)}{f_\theta(y)} \text{ is independent of }\theta,

with the equivalence understood up to null sets in the general measure-theoretic formulation. The criterion follows by considering likelihood ratios relative to a fixed parameter value: their collection generates the minimal sufficient sigma-field. Each ratio is measurable with respect to any sufficient sigma-field; conversely, the ratios themselves give a factorization and hence a sufficient sigma-field. The equivalence classes of samples with identical ratios are therefore exactly the classes retained by a minimal sufficient statistic.

For two samples \(x\) and \(y\), the terms in their Gaussian densities that involve only \(n,p\), and \(\Sigma\) cancel. The ratio is

\frac{f_{\mu,\Sigma}(x)}{f_{\mu,\Sigma}(y)} = \exp\!\left\{ -\frac12\operatorname{tr}\!\left[\Sigma^{-1}(Q(x)-Q(y))\right] +\mu'\Sigma^{-1}(s(x)-s(y)) \right\}.

If \(s(x)=s(y)\) and \(Q(x)=Q(y)\), this ratio equals one for every parameter value. In the reverse direction, suppose it is independent of \((\mu,\Sigma)\). Fix any positive-definite \(\Sigma\) and vary \(\mu\) over \(\mathbb R^p\). The coefficient of \(\mu\) must vanish, so \(\Sigma^{-1}(s(x)-s(y))=0\), which implies \(s(x)=s(y)\). The remaining ratio is independent of \(\Sigma\) only if \(\operatorname{tr}(\Sigma^{-1}D)=0\) for every positive-definite \(\Sigma\), where \(D=Q(x)-Q(y)\) is symmetric. Positive-definite matrices range over an open set. In particular, substituting \(A=I+tH\) for sufficiently small \(t\), with \(H\) any symmetric matrix, into \(\operatorname{tr}(AD)=0\) implies \(\operatorname{tr}(HD)=0\) for every symmetric \(H\). Taking \(H=D\) gives \(\operatorname{tr}(D^2)=0\), hence \(D=0\). The likelihood-ratio criterion now yields the result.

Theorem: The pair \((s,Q)\), or equivalently \((\bar X,C)\), is minimal sufficient for \((\mu,\Sigma)\) in the unrestricted multivariate Gaussian model.

Worked Example: Different Data With the Same Summary

In the univariate model, compare the ordered samples \(x=(-1,0,1)\) and \(y=(1,1,-2)/\sqrt{3}\), each of size three. Their sums are both zero. Their sums of squares are \(1+0+1=2\) and \((1+1+4)/3=2\), respectively.

For either sample the likelihood is therefore \((2\pi)^{-3/2}\sigma^{-3}\exp\{-(2+3\mu^2)/(2\sigma^2)\}\): the sum-of-squares term is \(2\), the cross-term vanishes because the sum is zero, and the mean term is \(3\mu^2\). The likelihoods agree for every \(\mu\) and \(\sigma^2>0\), although the observations are not the same. This illustrates that minimal sufficiency retains the likelihood-relevant summary, not the full sample.

Worked Example: Why the Sample Mean Alone Is Not Sufficient

Consider the univariate model with \(n=2\), and the samples \(x=(0,0)\) and \(y=(1,-1)\). Both have sample mean zero, but their sums of squares are \(0\) and \(2\).

Their density ratio is \[ \frac{f_{\mu,\sigma^2}(x)}{f_{\mu,\sigma^2}(y)} = \exp\!\left\{ -\frac{2\mu^2-(2+2\mu^2)}{2\sigma^2} \right\} =\exp(1/\sigma^2). \] It depends on the unknown variance. Thus the sample mean does not place these samples in the same likelihood-ratio class, and cannot by itself be sufficient when both mean and variance are unknown. The scatter information is needed as well.

What Sufficiency Does—and Does Not—Say

Sufficiency is a property of the likelihood and the model, not a claim that a particular estimator exists or behaves well. For example, the centered scatter \(C\) can be singular when there are too few observations to span all \(p\) dimensions. The pair \((\bar X,C)\) remains sufficient: the factorization and likelihood-ratio arguments did not require \(C\) to be invertible. Questions about existence of a covariance maximum-likelihood estimate are separate.

The result also depends on the model’s parameter space. If the covariance were known, or if it were restricted to a particular structure, the likelihood ratios could retain different information. Here the conclusion uses both the unrestricted mean and the unrestricted positive-definite covariance; it should not be transferred to a different Gaussian model without checking its likelihood.

Common Mistake

Wrong reasoning: “The sample mean is sufficient because the Gaussian likelihood contains the sample mean.” The likelihood also contains \(Q\), or equivalently the centered scatter. Two samples with the same mean can have different likelihood ratios as the covariance varies, as the preceding calculation demonstrates.

Correction: Inspect the full likelihood as a function of the unknown parameters. In this model its data-dependent terms involve both \(s\) and \(Q\). Factorization establishes sufficiency of their joint summary, and the likelihood-ratio criterion shows that neither can be discarded when both parameters are unknown.

Key takeaway: For independent multivariate normal observations with unknown mean and unrestricted covariance, the sample mean and centered scatter form a minimal sufficient statistic. The result follows by factoring the likelihood and showing that equal likelihood-ratio classes are exactly equal values of the two-part summary.

Check Your Understanding

Use the likelihood and the two criteria developed above to answer these questions.

Write \(Q\) in terms of the sample mean and centered scatter.
Why does the factorization theorem establish sufficiency of \((s,Q)\)?
In the likelihood-ratio argument, why can varying \(\mu\) force \(s(x)=s(y)\)?
What role does the assumption that \(\Sigma\) ranges over all positive-definite matrices play in identifying \(Q\)?
Does a singular centered scatter matrix prevent \((\bar X,C)\) from being sufficient? Explain.