Sufficient Statistics for Normal Mean and Variance

Why the Parameter Space Matters

A sufficient statistic retains all the sample information relevant to the parameter in a specified model. For a normal sample, the right summary depends on which parameters are unknown. If the variance is fixed, the likelihood uses the data through their sum; if the mean is fixed, it uses their squared deviations from that mean. When both are unknown, both kinds of information are needed.

Throughout, let \(X_1,\ldots,X_n\) be independent observations with \(X_i\sim N(\mu,\sigma^2)\), where \(n\) is fixed and \(\sigma^2>0\). The sample space is \(\mathbb{R}^n\). We consider three different parameter spaces: \(\mu\) unknown and \(\sigma^2\) known; \(\sigma^2\) unknown and \(\mu\) known; and both \(\mu\) and \(\sigma^2\) unknown.

Definition: A statistic \(T(X_1,\ldots,X_n)\) is sufficient for a parameter in a model if the conditional distribution of the full sample given \(T\) does not depend on that parameter. For a family of densities with a common dominating measure, the factorization criterion says that \(T\) is sufficient if the density can be written as \(f_\theta(x)=g_\theta(T(x))h(x)\), where \(g_\theta\) and \(h\) are nonnegative measurable functions and \(h\) does not depend on \(\theta\). The normal densities here are positive on the common support \(\mathbb{R}^n\), so the criterion applies.

The key is to factor the joint density, or equivalently the likelihood, while treating known parameters as fixed constants. A statistic sufficient in one of these models need not be sufficient in another: changing which parameters are unknown changes which data summaries can carry information about them.

Known Variance: The Sample Sum

Suppose \(\sigma^2\) is fixed and known, while \(\mu\) is unknown. The joint density at \(x=(x_1,\ldots,x_n)\) is

f_\mu(x) =(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right\}.

Expanding the squares gives \(\sum_i(x_i-\mu)^2=\sum_i x_i^2-2\mu\sum_i x_i+n\mu^2\). Therefore

f_\mu(x) = \underbrace{\exp\left\{\frac{\mu}{\sigma^2}\sum_{i=1}^n x_i -\frac{n\mu^2}{2\sigma^2}\right\}}_{g_\mu(\sum_i x_i)} \underbrace{(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n x_i^2\right\}}_{h(x)}.

The second factor may involve the observations, but it does not involve \(\mu\). By the factorization criterion, \(T=\sum_{i=1}^n X_i\) is sufficient for \(\mu\). Since \(n\) is fixed, the sample mean \(\bar X=T/n\) is an equivalent summary: either determines the other.

Worked Example: Known Variance and Two Samples With the Same Sum

Suppose \(n=3\) and \(\sigma^2=4\) is known. Compare the samples \(x=(1,2,4)\) and \(y=(0,3,4)\).

Both have sum \(7\): \(1+2+4=7\) and \(0+3+4=7\). Their sums of squares differ: \(1^2+2^2+4^2=21\), while \(0^2+3^2+4^2=25\). The likelihood for a sample \(z\) is proportional to \(\exp\{-[\sum_i z_i^2-2\mu\sum_i z_i+3\mu^2]/8\}\). Thus

\frac{L_x(\mu)}{L_y(\mu)} = \exp\left\{-\frac{(21-14+3\mu^2)-(25-14+3\mu^2)}{8}\right\} =\exp(1/2).

The ratio is independent of \(\mu\). The samples can have different likelihoods, but their likelihoods differ only by a factor that does not depend on the unknown mean. This is exactly the data separation expressed by the factorization: the likelihood's dependence on \(\mu\) is through the common sum \(7\).

Known Mean: Squared Deviations From That Mean

Now suppose \(\mu=\mu_0\) is fixed and known, while \(\sigma^2\) is unknown. Substituting the known mean into the joint density yields

f_{\sigma^2}(x) =(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2} \sum_{i=1}^n(x_i-\mu_0)^2\right\}.

All the data enter this expression through the single quantity \(T=\sum_i(X_i-\mu_0)^2\). More explicitly, take \(g_{\sigma^2}(t)=(2\pi\sigma^2)^{-n/2}\exp\{-t/(2\sigma^2)\}\) and \(h(x)=1\). This is the required factorization, so \(T\) is sufficient for \(\sigma^2\). The center \(\mu_0\) is the known mean, not an estimate from the data.

Worked Example: Known Mean and the Variance Summary

Let \(n=3\), with known mean \(\mu_0=2\), and observe \(x=(1,2,4)\). The squared deviations from the specified mean are

$$ (1-2)^2+(2-2)^2+(4-2)^2=1+0+4=5. $$

The likelihood as a function of the unknown variance is therefore

L(\sigma^2) =(2\pi\sigma^2)^{-3/2}\exp\left\{-\frac{5}{2\sigma^2}\right\}.

For another sample of size three with the same known mean, the likelihood has the same dependence on \(\sigma^2\) whenever its squared deviations from \(2\) also sum to \(5\). For instance, \(y=(3,2,0)\) gives \(1+0+4=5\) as well. This verifies directly that the likelihood uses these samples through the stated squared-deviation summary.

Both Mean and Variance Unknown: Two Summaries

When \(\mu\) and \(\sigma^2\) are both unknown, expansion of the same normal likelihood gives

f_{\mu,\sigma^2}(x) =(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{\sum_i x_i^2-2\mu\sum_i x_i+n\mu^2}{2\sigma^2}\right\}.

The data appear through both \(\sum_i x_i\) and \(\sum_i x_i^2\). In particular, the factorization can be written as

f_{\mu,\sigma^2}(x) = \underbrace{(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{\sum_i x_i^2-2\mu\sum_i x_i+n\mu^2}{2\sigma^2}\right\}}_{g_{\mu,\sigma^2}(\sum_i x_i,\sum_i x_i^2)} \underbrace{1}_{h(x)}.

Consequently, \(T=(\sum_i X_i,\sum_i X_i^2)\) is sufficient for the pair \((\mu,\sigma^2)\). A more interpretable equivalent pair is \((\bar X,\sum_i(X_i-\bar X)^2)\). To verify equivalence, write \(S_1=\sum_i X_i\) and \(S_2=\sum_i X_i^2\). Then \(\bar X=S_1/n\) and

\sum_i(X_i-\bar X)^2=S_2-\frac{S_1^2}{n}.

Conversely, \(S_1=n\bar X\) and \(S_2=\sum_i(X_i-\bar X)^2+n\bar X^2\). Each pair can therefore be calculated from the other without knowing either parameter.

Worked Example: Both Normal Parameters Unknown

For \(x=(1,2,4)\), calculate the two raw summaries: \(S_1=1+2+4=7\) and \(S_2=1^2+2^2+4^2=21\). The likelihood is

L(\mu,\sigma^2) =(2\pi\sigma^2)^{-3/2} \exp\left\{-\frac{21-14\mu+3\mu^2}{2\sigma^2}\right\}.

The equivalent centered summaries are \(\bar x=7/3\) and

\sum_i(x_i-\bar x)^2 =21-\frac{7^2}{3} =21-\frac{49}{3} =\frac{14}{3}.

The centered form checks the identity: \((1-7/3)^2+(2-7/3)^2+(4-7/3)^2=16/9+1/9+25/9=42/9=14/3\). The likelihood depends on the data through the two summaries, not through the order in which the three observations were recorded.

One Likelihood Calculation, Three Conclusions

The distinctions can be summarized by writing the unknown parameters beside the data summary that enters the likelihood. These are statements about the particular parameter space in each row; a parameter described as known is held fixed when applying the factorization criterion.

Model	Sufficient statistic from factorization	Why this summary appears
\(\mu\) unknown, \(\sigma^2\) known	\(\sum_i X_i\), equivalently \(\bar X\)	The unknown mean multiplies the sample sum in the expanded log-likelihood.
\(\mu=\mu_0\) known, \(\sigma^2\) unknown	\(\sum_i(X_i-\mu_0)^2\)	The unknown variance scales the sum of squared deviations from the fixed mean.
\(\mu\) and \(\sigma^2\) unknown	\((\sum_i X_i,\sum_i X_i^2)\), equivalently \((\bar X,\sum_i(X_i-\bar X)^2)\)	Both the first and second raw data summaries occur in the likelihood.

Common Mistake: Using the Wrong Center

A frequent error is to use \(\sum_i(X_i-\bar X)^2\) as the complete variance summary when the mean is known. That quantity is centered at the observed sample mean, whereas the likelihood with known mean \(\mu_0\) contains \(\sum_i(X_i-\mu_0)^2\). The two are related by

\sum_i(X_i-\mu_0)^2 = \sum_i(X_i-\bar X)^2+n(\bar X-\mu_0)^2.

The identity follows by writing \(X_i-\mu_0=(X_i-\bar X)+(\bar X-\mu_0)\), squaring, and summing; the cross term is zero because \(\sum_i(X_i-\bar X)=0\). Thus the known-mean likelihood uses a quantity that includes both the sample's spread around its own mean and the displacement of that mean from \(\mu_0\). Keeping only the first term generally discards information relevant to \(\sigma^2\).

The reverse mistake is to use only the sample sum when both parameters are unknown. The likelihood also contains \(\sum_i X_i^2\), so the sum alone does not provide the displayed factorization for the two-parameter model. The safe procedure is to expand the likelihood and inspect every data-dependent term before proposing a sufficient statistic.

Key takeaway: For an iid normal sample, factor the likelihood under the actual set of unknown parameters. The sample sum suffices for an unknown mean with known variance; squared deviations from the known mean suffice for an unknown variance; and the pair of first and second summaries suffices when both parameters are unknown.

Check Your Understanding

Use the likelihood factorization, not a memorized summary, to answer each question.

If \(\sigma^2\) is known and \(\mu\) is unknown, why are \(\sum_i X_i\) and \(\bar X\) interchangeable as sufficient statistics?
For known \(\mu_0\), write the data summary in the likelihood for an unknown \(\sigma^2\). Why is the center fixed at \(\mu_0\)?
When both normal parameters are unknown, express \(\sum_i X_i^2\) using \(\bar X\) and \(\sum_i(X_i-\bar X)^2\).
For the data \(2,3,5\) and known mean \(3\), calculate the sufficient squared-deviation summary.
Explain why a factorization that works when \(\sigma^2\) is known does not automatically establish sufficiency for a model where \(\sigma^2\) is unknown.