Gaussian Mean and Variance: A Likelihood Inference Workshop

Why Use a Gaussian Sample as a Likelihood Workshop?

The Gaussian model is simple enough to calculate with directly, yet it illustrates the central steps of likelihood inference: specify a model, maximize its likelihood, quantify curvature, and test a restriction. It also gives an unusually useful benchmark. When observations are independent and normally distributed, the usual Student t test and confidence interval have exact finite-sample distributions, so we can compare them with the asymptotic Wald, likelihood-ratio, and score procedures.

Let \(X_1,\ldots,X_n\) be independent observations from \(N(\mu,\sigma^2)\), where \(\mu\) is unrestricted and \(\sigma^2>0\). The variance is unknown and is a nuisance parameter when the main question concerns \(\mu\). All calculations below are conditional on this model; they do not make a causal interpretation of differences in means.

Model and assumptions: \(X_1,\ldots,X_n\) are independent and identically distributed as \(N(\mu,\sigma^2)\), with \(\mu\in\mathbb{R}\) and \(0<\sigma^2<\infty\). The observations are the data, and the parameter space for the variance excludes zero. Under these assumptions, for \(n\ge2\), the observations are not all identical with probability one.

Likelihood, Score, and Information

Write \(v=\sigma^2\) temporarily to make derivatives with respect to the variance clear. For observed values \(x_1,\ldots,x_n\), the likelihood is the product of the Gaussian densities. Its log, up to no omitted parameter-dependent terms, is

\ell(\mu,v) =-\frac{n}{2}\log(2\pi v) -\frac{1}{2v}\sum_{i=1}^n(x_i-\mu)^2.

The score is the vector of first derivatives of the log-likelihood. The observed information is the negative matrix of second derivatives; the expected information is its expectation under the model.

U_\mu(\mu,v)=\frac{1}{v}\sum_{i=1}^n(x_i-\mu), \qquad U_v(\mu,v)=-\frac{n}{2v} +\frac{1}{2v^2}\sum_{i=1}^n(x_i-\mu)^2.

Taking expectations of the negative second derivatives gives the expected information matrix, in the parameter order \((\mu,v)\):

I(\mu,v)= \begin{pmatrix} n/v & 0\\ 0 & n/(2v^2) \end{pmatrix}.

In particular, the expected cross-information is zero. In this model, the mean and variance parameters are orthogonal in the expected-information sense. This simplifies inference for the mean: its efficient information after accounting for the nuisance variance is \(n/v\), the same as its mean-parameter information before adjustment.

Worked Example: Fitting a Small Gaussian Sample

Suppose the hypothetical observations are \(8,10,11,11\). Their mean is \(\bar{x}=10\), and the sum of squared deviations from the mean is \( (8-10)^2+(10-10)^2+(11-10)^2+(11-10)^2=6\).

Setting the mean score to zero gives \(\hat{\mu}=\bar{x}=10\). Setting the variance score to zero at this mean gives \(\hat{v}=6/4=1.5\). The fitted standard deviation is therefore \(\sqrt{1.5}\), approximately \(1.225\) in the same units as the observations.

At \((\hat{\mu},\hat{v})=(10,1.5)\), both score components are zero. The expected information evaluated there is \(I(10,1.5)=\begin{pmatrix}4/1.5&0\\0&4/(2(1.5)^2)\end{pmatrix}\), or approximately \(\begin{pmatrix}2.667&0\\0&0.8889\end{pmatrix}\). The inverse information gives the usual large-sample covariance estimate; in particular, the estimated variance of \(\hat{\mu}\) is \(1.5/4=0.375\).

Maximum Likelihood Estimates and a Necessary Qualification

For any fixed \(v\), the log-likelihood is maximized over \(\mu\) at \(\bar{x}\), because the sum of squared deviations is minimized there. Substituting this value into the likelihood leaves a function of \(v\) alone. Differentiating that profile log-likelihood and setting its derivative to zero gives \(v=\sum_i(x_i-\bar{x})^2/n\).

\hat{\mu}=\bar{x}, \qquad \hat{\sigma}^2_{\mathrm{MLE}} =\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2, \quad\text{when }\sum_{i=1}^n(x_i-\bar{x})^2>0.

The qualification matters. If every observed value is identical, the residual sum of squares is zero, and the displayed variance estimate would be outside the parameter space. In that case there is no maximum likelihood estimate in the stated parameter space: taking \(\mu\) equal to the common observation and letting \(v\) approach zero makes the likelihood unbounded. For a Gaussian sample with \(n\ge2\), this event has probability zero, but it remains a possible data configuration that must be handled correctly.

The maximum likelihood variance uses divisor \(n\). The sample variance \(s^2=\sum_i(x_i-\bar{x})^2/(n-1)\) uses divisor \(n-1\) and is unbiased for \(\sigma^2\) under this model. These are different estimates, serving different purposes; substituting one for the other changes finite-sample formulas.

Three Asymptotic Tests of a Mean

Consider testing \(H_0:\mu=\mu_0\) against a two-sided alternative \(H_a:\mu\ne\mu_0\), with \(\sigma^2\) unknown. Under the regularity conditions for likelihood-based inference, each of the following statistics has an approximate chi-square distribution with one degree of freedom under \(H_0\), when the sample is sufficiently informative. A level-\(\alpha\) test rejects for a statistic exceeding the \(1-\alpha\) quantile of that reference distribution.

Wald: measures the squared distance between the unrestricted estimate and the null value, scaled by the estimated variance of the estimate.
Likelihood ratio: compares the maximized likelihood under the null restriction with the unrestricted maximized likelihood.
Score: measures the score at the restricted estimate, scaled by the information under the null.

Let \(R=\sum_i(x_i-\bar{x})^2\), \(\delta=\bar{x}-\mu_0\), and \(R_0=\sum_i(x_i-\mu_0)^2\). The identity \(R_0=R+n\delta^2\) follows by expanding \(x_i-\mu_0=(x_i-\bar{x})+\delta\), since \(\sum_i(x_i-\bar{x})=0\). If \(R>0\), profiling out the variance under each model yields

W=\frac{n\delta^2}{\hat{\sigma}^2_{\mathrm{MLE}}} =\frac{n^2\delta^2}{R}, \qquad LR=n\log\!\left(\frac{R_0}{R}\right), \qquad S=\frac{n^2\delta^2}{R_0}.

For the score formula, the restricted variance estimate is \(\hat{v}_0=R_0/n\). The mean score at the restricted estimate is \(U_\mu=n\delta/\hat{v}_0\); its information is \(n/\hat{v}_0\). Because the expected cross-information is zero, the nuisance-adjusted score statistic is \(U_\mu^2/(n/\hat{v}_0)=n\delta^2/\hat{v}_0=n^2\delta^2/R_0\). For the likelihood-ratio statistic, the maximized Gaussian log-likelihood under each fit differs only through its profiled residual sum of squares, giving \(2(\ell_{\mathrm{unrestricted}}-\ell_{\mathrm{null}})=n\log(R_0/R)\).

Worked Example: Comparing the Three Tests

Use the hypothetical observations \(7,8,9,9,10,11\) to test \(H_0:\mu=8\) against \(H_a:\mu\ne8\). Here \(n=6\), \(\bar{x}=9\), and \(R=4+1+0+0+1+4=10\). Thus \(\delta=1\), while \(R_0=R+n\delta^2=10+6=16\).

The Wald statistic is \(W=6^2(1)^2/10=3.600\). The likelihood-ratio statistic is \(LR=6\log(16/10)=6\log(1.6)\approx2.820\). The score statistic is \(S=6^2(1)^2/16=2.250\). All three are compared with a \(\chi^2_1\) reference distribution for their asymptotic p-values.

Those approximate two-sided p-values are about 0.058 for Wald, 0.093 for the likelihood ratio, and 0.134 for the score test. At \(\alpha=0.05\), none rejects using these asymptotic calibrations. They differ because they scale the observed departure from the null in different ways; finite-sample differences can be noticeable.

The Exact Gaussian Benchmark

Under the stated independent Gaussian model and \(H_0\), the statistic \(T=(\bar{X}-\mu_0)/(s/\sqrt{n})\) has exactly a Student t distribution with \(n-1\) degrees of freedom. The reason is that \(\sqrt{n}(\bar{X}-\mu_0)/\sigma\) is standard normal, \((n-1)s^2/\sigma^2\) has a chi-square distribution with \(n-1\) degrees of freedom, and these two quantities are independent. Their ratio, with the variance estimate in the denominator, has the \(t_{n-1}\) distribution. This exact result depends on normality; it is not merely a large-sample approximation.

The relationship to the three likelihood statistics can be made explicit. Since \(s^2=R/(n-1)\), \(T^2=(n-1)n\delta^2/R=(n-1)q\), where \(q=n\delta^2/R\). Consequently, for a nondegenerate sample,

W=nq=\frac{n}{n-1}T^2, \qquad LR=n\log(1+q)=n\log\!\left(1+\frac{T^2}{n-1}\right), \qquad S=\frac{nq}{1+q} =\frac{nT^2}{n-1+T^2}.

Each has an asymptotic \(\chi^2_1\) reference distribution, but none is generally equal to \(T^2\) at finite \(n\). For an exact level-\(\alpha\) test of this Gaussian mean, compare \(|T|\) with a \(t_{n-1}\) critical value, or equivalently compare \(T^2\) with the corresponding squared critical value. This provides a finite-sample benchmark against which the asymptotic likelihood tests can be assessed.

Worked Example: An Exact Confidence Interval

For the same six hypothetical observations, \(s^2=R/(n-1)=10/5=2\), so \(s=\sqrt{2}\approx1.414\). An exact 95% confidence interval for \(\mu\), under the Gaussian model, is \(\bar{x}\pm t_{0.975,5}s/\sqrt{n}\), with \(t_{0.975,5}\approx2.5706\).

Substitution gives \(9\pm2.5706\sqrt{2}/\sqrt{6}=9\pm1.484\), to three decimal places. The interval is approximately \((7.516,\ 10.484)\) in the units of the observations. Its 95% coverage means that this procedure, applied repeatedly to independent samples from the same Gaussian population, contains the fixed mean in 95% of samples. It does not mean that the parameter has a 95% probability of lying in this particular computed interval.

For \(H_0:\mu=8\), the corresponding exact test statistic is \(T=(9-8)/\sqrt{2/6}=\sqrt{3}\approx1.732\), with \(df=5\). Its two-sided p-value is approximately 0.144, consistent with the interval containing 8. The asymptotic likelihood p-values above are not the exact finite-sample p-value.

Interpretation and Common Pitfalls

The three likelihood tests are useful because they extend to models where exact Gaussian calculations are unavailable. Their asymptotic reference distributions make them broadly applicable, but the Gaussian benchmark shows why small-sample calibration deserves attention. A p-value measures how unusual a statistic at least as extreme would be under the null and the assumed model; it is not the probability that \(H_0\) is true. Likewise, failure to reject is not proof that the null is correct.

Confusing the two variance estimates. The Gaussian MLE uses divisor \(n\); the sample variance \(s^2\) uses \(n-1\). Use the MLE in the displayed likelihood-based Wald formula and use \(s\) in the exact t statistic.
Assuming the variance MLE always exists. If all observations are identical, the likelihood is unbounded as the variance approaches zero, so no MLE exists when the parameter space requires positive variance.
Treating asymptotic calibration as exact. A \(\chi^2_1\) reference is generally approximate at finite sample sizes. Under the Gaussian model, the t test supplies an exact benchmark for the mean.
Reading different statistics as contradictory evidence. Wald, likelihood-ratio, and score statistics use different evaluations of the likelihood. They are asymptotically equivalent under regularity conditions, not necessarily numerically identical in a small sample.

Key takeaway: In the Gaussian location-scale model, the likelihood gives a direct route from the score and information to three asymptotic tests of the mean. The exact Student t result remains the appropriate finite-sample benchmark when the independent Gaussian assumptions hold.

Check Your Understanding

Try these questions using the definitions and calculations above.

For a sample with positive residual sum of squares \(R\), what are the maximum likelihood estimates of the mean and variance?
Why does a data set with all observations identical have no variance MLE when the parameter space requires \(\sigma^2>0\)?
In testing \(H_0:\mu=\mu_0\), what is the restricted variance estimate used in the score test?
State the exact reference distribution and degrees of freedom for the Gaussian t statistic for the mean.
Why can the Wald, likelihood-ratio, and score p-values differ from the exact t-test p-value in a small sample?