Joint MLEs for a Normal Mean and Variance

Why Estimate the Mean and Variance Together?

When the mean and variance of a normal population are both unknown, each affects the likelihood of the observed sample. The mean determines where the normal density is centered; the variance determines its spread. Estimating one while treating the other as known does not solve the joint estimation problem. The maximum likelihood approach instead chooses the pair of parameter values that makes the observed sample most likely under the model.

A key result is easy to misremember: the maximum likelihood estimate of the variance divides the sum of squared deviations from the sample mean by \(n\), not by \(n-1\). The \(n-1\) denominator gives the familiar unbiased sample variance. These estimates answer different questions and are obtained by different criteria.

Model and assumptions: Let \(X_1,\ldots,X_n\) be independent and identically distributed observations from \(N(\mu,\sigma^2)\), where both \(\mu\) and \(\sigma^2\) are unknown and \(\sigma^2>0\). The observed values are \(x_1,\ldots,x_n\). The likelihood is the joint density of these observations, viewed as a function of the parameters.

Likelihood and Score Equations

Independence means the joint density is the product of the individual normal densities. Define the sample mean by \(\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i\). The likelihood and log-likelihood are

L(\mu,\sigma^2) =(2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right), \qquad \sigma^2>0.

\ell(\mu,\sigma^2) =-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2.

For an interior maximum, differentiate the log-likelihood with respect to each unknown parameter and set both derivatives to zero. Write \(Q(\mu)=\sum_{i=1}^n(x_i-\mu)^2\). The score equations are

\frac{\partial \ell}{\partial \mu} =\frac{1}{\sigma^2}\sum_{i=1}^n(x_i-\mu)=0, \qquad \frac{\partial \ell}{\partial \sigma^2} =-\frac{n}{2\sigma^2}+\frac{Q(\mu)}{2(\sigma^2)^2}=0.

The first equation implies \(n\mu=\sum_{i=1}^n x_i\), so the candidate mean is \(\bar{x}\). Substituting this into the second equation gives the candidate variance.

\widehat{\mu}_{\mathrm{MLE}}=\bar{x}, \qquad \widehat{\sigma}^2_{\mathrm{MLE}} =\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2.

The equation for the variance is solved by multiplying by \(2(\sigma^2)^2\): \(-n\sigma^2+Q(\mu)=0\), hence \(\sigma^2=Q(\mu)/n\). This candidate must still be checked as a maximum and must lie in the parameter space \(\sigma^2>0\).

Why These Candidates Maximize the Likelihood

A score equation alone identifies a stationary candidate; it does not establish a maximum. Here the structure of the squared deviations gives a direct global check. For every \(\mu\),

Q(\mu) =\sum_{i=1}^n(x_i-\bar{x})^2+n(\mu-\bar{x})^2.

To verify the identity, write \(x_i-\mu=(x_i-\bar{x})+(\bar{x}-\mu)\), square and sum. The cross term is zero because \(\sum_{i=1}^n(x_i-\bar{x})=0\). Therefore \(Q(\mu)\) is smallest at \(\mu=\bar{x}\).

For any fixed \(\mu\) with \(Q(\mu)>0\), the log-likelihood as a function of \(s=\sigma^2>0\), apart from a constant, is \(-\frac{n}{2}\log s-\frac{Q(\mu)}{2s}\). Its derivative is \(-n/(2s)+Q(\mu)/(2s^2)\), which is positive when \(s<Q(\mu)/n\) and negative when \(s>Q(\mu)/n\). Thus it reaches its maximum at \(s=Q(\mu)/n\). At that maximizing value, the log-likelihood decreases as \(Q(\mu)\) increases: its parameter-dependent part becomes \(-\frac{n}{2}\log(Q(\mu)/n)-\frac{n}{2}\). Since \(Q(\mu)\) is minimized at \(\bar{x}\), the joint maximum occurs at the two estimates above, provided \(Q(\bar{x})>0\).

Theorem: If the sample contains at least two different observed values, then \(Q(\bar{x})>0\), and the joint normal likelihood has a finite maximum at \(\widehat{\mu}_{\mathrm{MLE}}=\bar{x}\) and \(\widehat{\sigma}^2_{\mathrm{MLE}}=Q(\bar{x})/n\). If all observed values are identical, \(Q(\bar{x})=0\); as \(\sigma^2\) approaches zero, the likelihood is unbounded, so no finite MLE exists when the parameter space requires \(\sigma^2>0\).

MLE Variance Versus Unbiased Sample Variance

The MLE variance and the unbiased sample variance use the same squared deviations but different denominators. For \(n\geq2\), the unbiased sample variance is conventionally

S^2=\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})^2.

The reason for \(n-1\) can be seen from an expectation calculation under the stated independent normal model. The identity \(\sum_{i=1}^n(X_i-\bar{X})^2=\sum_{i=1}^n(X_i-\mu)^2-n(\bar{X}-\mu)^2\) holds for every sample. Taking expectations, the first sum has expectation \(n\sigma^2\). Since \(\operatorname{Var}(\bar{X})=\sigma^2/n\), the expectation of the subtracted term is \(n(\sigma^2/n)=\sigma^2\). Consequently,

E\left[\sum_{i=1}^n(X_i-\bar{X})^2\right] =n\sigma^2-\sigma^2=(n-1)\sigma^2, \qquad E[S^2]=\sigma^2.

Thus \(S^2\) is unbiased for the population variance, while the MLE variance has expectation \(\frac{n-1}{n}\sigma^2\). The MLE is not required to be unbiased: it is defined by maximizing the likelihood. If estimating the standard deviation itself, the corresponding MLE is \(\widehat{\sigma}_{\mathrm{MLE}}=\sqrt{\widehat{\sigma}^2_{\mathrm{MLE}}}\); it is not obtained by maximizing a separate likelihood for a different model.

Worked Example: Joint Estimates From Four Observations

A hypothetical laboratory records measurements \(3,5,7,9\) from four independent observations in a normal population with unknown mean and variance. Find the joint MLEs, the unbiased sample variance, and check the score equations. The method is to compute the sample mean and the sum of squared deviations, then apply the joint normal MLE formulas.

Step 1: Find the sample mean. Here \(n=4\), and the sum is \(3+5+7+9=24\), so \(\bar{x}=24/4=6\).

Step 2: Compute the squared deviations. The deviations are \(-3,-1,1,3\); their squares are \(9,1,1,9\). Therefore \(Q(\bar{x})=9+1+1+9=20\).

Step 3: Calculate both variance estimates. The joint maximum likelihood estimates are \(\widehat{\mu}_{\mathrm{MLE}}=6\) and \(\widehat{\sigma}^2_{\mathrm{MLE}}=20/4=5\). The unbiased sample variance is \(S^2=20/(4-1)=20/3\).

Step 4: Check the scores. At \(\mu=6\), the sum of deviations is \(-3-1+1+3=0\), so the mean score is zero. At \(\sigma^2=5\), the variance score is \(-4/(2\cdot5)+20/(2\cdot25)=-0.4+0.4=0\).

The joint MLE is \((6,5)\), while the unbiased variance estimate is \(20/3\). The observations are not all identical, so the finite-maximum condition is satisfied. The common mistake avoided here is replacing the likelihood denominator \(n=4\) with the unbiased-variance denominator \(n-1=3\).

Worked Example: Using a Frequency Table

A hypothetical quality audit records the values \(1,3,5\) with frequencies \(2,3,1\), respectively. Treat the six observations as independent draws from a normal distribution with unknown mean and variance. Find the MLEs and the unbiased sample variance without expanding the frequency table unnecessarily.

Step 1: Find the sample size and sum. The sample size is \(n=2+3+1=6\). The sum of observations is \(2(1)+3(3)+1(5)=2+9+5=16\), so \(\bar{x}=16/6=8/3\).

Step 2: Calculate the squared-deviation sum. The sum of squared observations is \(2(1^2)+3(3^2)+1(5^2)=2+27+25=54\). Using \(Q(\bar{x})=\sum x_i^2-n\bar{x}^2\),

Q(\bar{x})=54-6\left(\frac{8}{3}\right)^2 =54-\frac{384}{9} =54-\frac{128}{3} =\frac{34}{3}.

Step 3: Calculate the estimates. The variance MLE is \((34/3)/6=34/18=17/9\), and the mean MLE is \(8/3\). The unbiased sample variance is \((34/3)/5=34/15\).

As a check, the deviations from \(8/3\) are \(-5/3\) twice, \(1/3\) three times, and \(7/3\) once. Their squared sum is \(2(25/9)+3(1/9)+49/9=102/9=34/3\), as calculated. The common mistake avoided is dividing the MLE squared-deviation sum by the number of distinct values, three, rather than by the total sample size, six.

Worked Example: Checking a Joint Candidate Against Another Parameter Pair

A hypothetical sample consists of \(1,2,3,4,5\), modeled as independent observations from \(N(\mu,\sigma^2)\) with both parameters unknown. Find the joint MLEs and compare their log-likelihood with that at the alternative pair \((\mu,\sigma^2)=(4,4)\). This comparison illustrates why the two unknowns must be assessed together.

Step 1: Find the joint MLE. The sample size is \(5\), the sum is \(15\), and \(\bar{x}=15/5=3\). The squared deviations are \(4,1,0,1,4\), so \(Q(\bar{x})=10\). Therefore \(\widehat{\mu}_{\mathrm{MLE}}=3\) and \(\widehat{\sigma}^2_{\mathrm{MLE}}=10/5=2\).

Step 2: Evaluate the log-likelihood at the MLE. Substituting \(n=5\), \(\mu=3\), and \(\sigma^2=2\) into the log-likelihood gives \(\ell(3,2)=-\frac{5}{2}\log(4\pi)-\frac{10}{4}=-\frac{5}{2}\log(4\pi)-2.5\).

Step 3: Evaluate it at the alternative pair. At \(\mu=4\), the squared deviations are \(9,4,1,0,1\), summing to \(15\). Thus \(\ell(4,4)=-\frac{5}{2}\log(8\pi)-\frac{15}{8}\).

Step 4: Compare. Subtracting, \(\ell(3,2)-\ell(4,4)=\frac{5}{2}\log 2-\frac{5}{8}\), which is approximately \(1.1079>0\). The MLE pair has the larger log-likelihood. The observations are not all identical, so the global-maximum result also applies. The common mistake avoided is optimizing the mean and variance separately while failing to evaluate whether a proposed pair is jointly better.

Common Mistakes and Practical Checks

Using \(n-1\) in the likelihood estimate. The variance MLE divides by \(n\); use \(n-1\) only for the unbiased sample variance.
Confusing variance and standard deviation. The MLE of the variance is \(Q(\bar{x})/n\); the MLE of the standard deviation is its square root.
Assuming every score solution is a maximum. Check the parameter space and the likelihood’s behavior. In particular, identical observations make the likelihood unbounded as the variance approaches zero.
Ignoring the model assumptions. The product likelihood and the expectation calculations here use independent observations from one normal distribution with a common mean and variance.

For an exam calculation, record \(n\), \(\bar{x}\), and \(Q(\bar{x})\) before substituting. This keeps the two denominators visible and makes it easier to detect arithmetic errors. The joint MLEs are useful because they provide the likelihood-maximizing parameter pair; the unbiased variance estimate is useful when unbiasedness is the criterion. They are both legitimate summaries, but they are not interchangeable.

Check Your Understanding

For each independent normal sample, determine the requested estimate. Assume the mean and variance are unknown unless a question says otherwise.

For observations \(2,4,6\), find the joint MLE of the mean and variance and the unbiased sample variance. (Answer: \(\widehat{\mu}_{\mathrm{MLE}}=4\), \(\widehat{\sigma}^2_{\mathrm{MLE}}=8/3\), and \(S^2=4\).)
A sample has \(n=8\), \(\sum x_i=40\), and \(\sum x_i^2=240\). Find the joint MLEs of the mean and variance. (Answer: \(\widehat{\mu}_{\mathrm{MLE}}=5\) and \(\widehat{\sigma}^2_{\mathrm{MLE}}=5\).)
Five observations are all equal to \(7\). Does a finite MLE of the variance exist when the parameter space is \(\sigma^2>0\)? (Answer: No; the likelihood is unbounded as \(\sigma^2\) approaches zero.)
A sample of size \(6\) has sample mean \(10\) and squared-deviation sum \(\sum_{i=1}^6(x_i-\bar{x})^2=30\). Find the variance MLE and the unbiased sample variance. (Answer: \(\widehat{\sigma}^2_{\mathrm{MLE}}=5\) and \(S^2=6\).)
A nonconstant sample of size \(10\) has squared-deviation sum \(27\). A candidate reports \(27/9\) as the maximum likelihood estimate of the variance. Correct the report and state what \(27/9\) estimates. (Answer: The variance MLE is \(27/10=2.7\); \(27/9=3\) is the unbiased sample variance.)

What You'll Learn