A Worked MLE Workshop for Continuous Samples

Why Continuous-Sample MLEs Need More Than a Score Equation

For a continuous model, a maximum likelihood estimate is a parameter value that maximizes the joint density of the observed sample, viewed as a function of the unknown parameter. Differentiating the log-likelihood is often an efficient way to locate a maximum, but it is not a complete method by itself. A score equation can have no solution, a solution outside the parameter space, or a stationary point that is not the global maximum. In models whose support depends on the parameter, the likelihood can also change abruptly at a boundary.

The calculations below use independent, identically distributed observations \(X_1,\ldots,X_n\), unless a model is stated otherwise. Write the observed values as \(x_1,\ldots,x_n\). The likelihood is the product of the sampling densities, evaluated at those observations. The central checks are: identify the parameter space, retain every support condition, solve any score equation, and compare the candidate with relevant boundaries or limiting behavior.

Definition: For a sample with joint density \(L(\theta;x_1,\ldots,x_n)\), an MLE is a parameter value \(\hat{\theta}\) in the stated parameter space that maximizes \(L\) over that space. Where the likelihood is positive, maximizing the log-likelihood gives the same maximizing parameter value. A score equation is a necessary condition for an interior maximum under differentiability; it does not by itself establish that an MLE exists or that a candidate is a maximum.

Exponential Samples: A Score Solution With a Domain Check

Suppose \(X_1,\ldots,X_n\) are independent exponential waiting times with rate \(\lambda>0\), measured in a specified time unit. Their density is \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x\geq 0\), and zero otherwise. For observed nonnegative values, the likelihood and log-likelihood are

L(\lambda)=\lambda^n e^{-\lambda\sum_{i=1}^n x_i}, \qquad \ell(\lambda)=n\log\lambda-\lambda\sum_{i=1}^n x_i,\quad \lambda>0.

If \(\sum x_i>0\), differentiating gives \(\ell'(\lambda)=n/\lambda-\sum x_i\). The score equation has the candidate \(\hat{\lambda}=n/\sum x_i\). The second derivative is \(-n/\lambda^2<0\), and the likelihood tends to zero as \(\lambda\) approaches zero or infinity when \(\sum x_i>0\). Thus the candidate is the global maximum. If every observation is zero, however, \(L(\lambda)=\lambda^n\) grows without bound as \(\lambda\to\infty\); there is then no finite MLE in the parameter space \(\lambda>0\).

Worked Example: Estimating an Exponential Rate

A hypothetical monitoring system records four independent waiting times, in days: \(0.5,\ 1,\ 1.5,\ 2\). Assume an exponential model with unknown rate \(\lambda>0\), in inverse days. Find the MLE and verify that it maximizes the likelihood. The method is the score equation followed by a boundary check; the mistake to avoid is treating the estimate as a mean rather than a rate.

Step 1: Form the likelihood. The sample size is \(n=4\), and the sum is \(0.5+1+1.5+2=5\) days. Independence gives

L(\lambda)=\lambda^4e^{-5\lambda}, \qquad \ell(\lambda)=4\log\lambda-5\lambda.

Step 2: Solve the score equation. The score is \(4/\lambda-5\). Setting it equal to zero gives \(4/\lambda=5\), so \(\hat{\lambda}=4/5=0.8\) per day.

Step 3: Verify the maximum. The second derivative is \(-4/\lambda^2<0\), and the likelihood tends to zero as \(\lambda\downarrow0\) and as \(\lambda\to\infty\). Therefore the score solution is the global maximum. Substitution gives \(L(0.8)=0.8^4e^{-4}=0.4096e^{-4}\), approximately \(0.0075\).

Check. The fitted exponential mean waiting time is \(1/\hat{\lambda}=1.25\) days, equal to the observed sample mean \(5/4\) days. This is a useful arithmetic check, not a replacement for maximizing the likelihood.

Normal Samples: Jointly Estimating the Mean and Variance

Now suppose \(X_1,\ldots,X_n\) are independent with \(X_i\sim N(\mu,\sigma^2)\), where \(\mu\) is unrestricted and \(\sigma^2>0\). The variance, not the standard deviation, is the second parameter. For observed real values, the log-likelihood is

\ell(\mu,\sigma^2) =-\frac{n}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2.

For fixed \(\sigma^2\), maximizing this expression in \(\mu\) is equivalent to minimizing the sum of squared deviations. The minimum occurs at \(\bar{x}=\sum x_i/n\). Differentiating with respect to \(\sigma^2\), and then substituting \(\mu=\bar{x}\), gives the variance candidate. This derivation also shows why one must check whether that candidate lies in the parameter space.

Maximize in the mean.
The derivative in \(\mu\) is \(\sum_i(x_i-\mu)/\sigma^2\). It equals zero at \(\mu=\bar{x}\); the sum of squares is minimized there.

Maximize in the variance.
Let \(S=\sum_i(x_i-\bar{x})^2\). After substituting \(\bar{x}\), the log-likelihood is \(-\frac n2\log(2\pi\sigma^2)-S/(2\sigma^2)\). Its derivative in \(\sigma^2\) is \(-n/(2\sigma^2)+S/(2(\sigma^2)^2)\), which vanishes at \(\sigma^2=S/n\) when \(S>0\).

Check existence and maximality.
When \(S>0\), the profile log-likelihood tends to negative infinity as \(\sigma^2\downarrow0\) or \(\sigma^2\to\infty\), and its derivative changes from positive to negative at \(S/n\). This gives the global maximum. When \(S=0\), the candidate variance is zero, which is not allowed; the likelihood is unbounded as \(\sigma^2\downarrow0\) at \(\mu=\bar{x}\), so no MLE exists.

Theorem: For an independent normal sample with both \(\mu\) and \(\sigma^2>0\) unknown, if \(S=\sum_{i=1}^n(x_i-\bar{x})^2>0\), the joint MLEs are \(\hat{\mu}=\bar{x}\) and \(\hat{\sigma}^2=S/n\). If all observations are equal, so \(S=0\), there is no MLE in the stated parameter space: at \(\mu=\bar{x}\), the likelihood tends to infinity as \(\sigma^2\downarrow0\).

Worked Example: Joint Normal Mean and Variance MLEs

A hypothetical laboratory records four independent measurements, modeled as \(N(\mu,\sigma^2)\) with both parameters unknown and \(\sigma^2>0\): \(2,\ 4,\ 4,\ 6\). Find the joint MLEs. Use the joint normal likelihood; the mistake to avoid is dividing the squared-deviation total by \(n-1\), which does not maximize this likelihood.

Step 1: Calculate the sample mean. Here \(n=4\), the sum is \(2+4+4+6=16\), and \(\bar{x}=16/4=4\).

Step 2: Calculate the squared-deviation total. The deviations are \(-2,0,0,2\), so \(S=(-2)^2+0^2+0^2+2^2=4+0+0+4=8\). Since \(S>0\), the normal MLEs exist.

Step 3: Apply the joint MLE formulas. The mean estimate is \(\hat{\mu}=4\), and the variance estimate is \(\hat{\sigma}^2=S/n=8/4=2\). The corresponding standard deviation estimate, if needed, is \(\sqrt{2}\).

Step 4: Check the likelihood value. At these estimates,

L(4,2)=(2\pi\cdot2)^{-4/2} \exp\left(-\frac{8}{2\cdot2}\right) =(4\pi)^{-2}e^{-2} =\frac{e^{-2}}{16\pi^2}.

The result uses divisor \(n=4\), not \(n-1=3\). Using \(8/3\) would give a different variance estimate and would not solve the likelihood score equation.

Uniform Samples: The Support Can Determine the Estimate

For a uniform model, the likelihood often contains an indicator for which parameter values could have generated the observed data. That condition must remain in the likelihood throughout maximization. A derivative of the density’s parameter-dependent factor alone cannot handle the jump from likelihood zero to positive likelihood at a support boundary.

First let \(X_1,\ldots,X_n\) be independent uniform observations on \([0,\theta]\), with \(\theta>0\). Put \(x_{(n)}=\max_i x_i\). For nonnegative observations, the likelihood is \(\theta^{-n}\) when \(\theta\geq x_{(n)}\), and zero otherwise. If \(x_{(n)}>0\), then \(\theta^{-n}\) decreases as \(\theta\) increases, so its maximum over the feasible parameter values occurs at the boundary \(\hat{\theta}=x_{(n)}\). If all observations are zero, the likelihood is \(\theta^{-n}\) for every \(\theta>0\), which is unbounded as \(\theta\downarrow0\); no MLE exists in that case.

Worked Example: A Uniform Upper-Bound MLE

A hypothetical device produces four independent readings uniformly distributed on \([0,\theta]\), where \(\theta>0\) is unknown. The observed readings are \(2.1,\ 3.4,\ 4.0,\ 3.1\). Find the MLE. The method is to maximize over the support-feasible values; the mistake to avoid is differentiating \(-n\log\theta\) while ignoring that the likelihood is zero for inadmissible \(\theta\).

Step 1: Find the support constraint. The largest reading is \(x_{(n)}=4.0\). All four observations are within the model’s support only if \(\theta\geq4.0\).

Step 2: Maximize over feasible values. For \(\theta\geq4.0\), the likelihood is \(L(\theta)=\theta^{-4}\). This decreases as \(\theta\) increases, so its maximum is at the smallest feasible value, \(\hat{\theta}=4.0\).

Step 3: Check. The maximum likelihood value is \(4^{-4}=1/256\). If \(\theta<4\), the likelihood is zero; if \(\theta>4\), it is smaller than \(1/256\). Thus the boundary value is the global maximum.

The same support-based reasoning applies when both endpoints of a uniform interval are unknown. Suppose the model is uniform on \([\alpha,\beta]\), with \(\alpha<\beta\). For a sample with minimum \(x_{(1)}\) and maximum \(x_{(n)}\), a positive likelihood requires \(\alpha\leq x_{(1)}\) and \(\beta\geq x_{(n)}\). On that feasible region the likelihood is \((\beta-\alpha)^{-n}\). When the observed range is positive, the interval width is smallest at \(\alpha=x_{(1)}\), \(\beta=x_{(n)}\), giving the maximum likelihood. This is a boundary maximum, not an interior score solution.

Worked Example: Estimating Both Uniform Endpoints

Four independent observations are modeled as uniform on an unknown interval \([\alpha,\beta]\), where \(\alpha<\beta\). The observations are \(5,\ 7,\ 6,\ 9\). Find both MLEs and verify the support conditions. The method is to minimize the feasible interval width; the mistake to avoid is using the sample average as an endpoint estimate.

Step 1: Identify the observed extremes. The sample minimum is \(x_{(1)}=5\) and the maximum is \(x_{(n)}=9\). Positive likelihood requires \(\alpha\leq5\) and \(\beta\geq9\).

Step 2: Maximize the likelihood. For feasible endpoints, \(L(\alpha,\beta)=(\beta-\alpha)^{-4}\). The width satisfies \(\beta-\alpha\geq9-5=4\). Since the likelihood decreases as the width increases, its maximum occurs at the smallest feasible width: \(\hat{\alpha}=5\) and \(\hat{\beta}=9\).

Step 3: Check. The fitted interval includes every observation, has width \(9-5=4>0\), and gives likelihood \(4^{-4}=1/256\). Any feasible interval with a larger width has a smaller likelihood.

Common Mistakes and a Practical Maximization Checklist

These models illustrate three different outcomes: an interior score solution for the exponential rate, joint interior estimates for a nondegenerate normal sample, and support-determined boundary estimates for uniform models. A careful solution checks the likelihood on its full domain rather than treating differentiation as the whole task.

Ignoring the parameter space. In the normal model, \(S/n=0\) is not an admissible variance when \(\sigma^2>0\). If all observations are equal, do not report zero as the MLE; the likelihood is unbounded as the variance approaches zero.
Dropping a support indicator. For \(U(0,\theta)\), the likelihood is zero when \(\theta\) is smaller than the sample maximum. A derivative that omits this condition cannot identify the boundary maximum.
Using a familiar estimator without checking the likelihood. The normal variance MLE uses divisor \(n\). In the worked sample, the incorrect calculation \(8/(4-1)=8/3\) is not the likelihood maximum; the score equation gives \(8/4=2\).
Forgetting limiting cases. If every exponential observation is zero, the likelihood grows without bound as the rate increases. If every \(U(0,\theta)\) observation is zero, it grows without bound as \(\theta\) decreases to zero. These cases do not have finite MLEs in the stated parameter spaces.

Key takeaway: Build the likelihood with its support conditions, solve score equations only for possible interior maxima, and then check boundaries and limiting behavior. A reported MLE must belong to the parameter space and actually maximize the likelihood. These methods are aligned with the SOA VEE Mathematical Statistics syllabus.

Check Your Understanding

For each problem, identify the relevant likelihood method and check that the proposed estimate is valid for the stated model.

Three independent waiting times, measured in hours, are modeled as exponential with unknown rate \(\lambda>0\). The observed values are \(2,\ 3,\ 5\). Find the rate MLE, including its units. (Answer: \(\hat{\lambda}=0.3\) per hour.)
Four independent waiting times, measured in hours, follow an exponential model with unknown positive rate. The observations are \(1,\ 2,\ 3,\ 4\). Find the rate MLE and the fitted mean waiting time. (Answer: \(\hat{\lambda}=0.4\) per hour and fitted mean \(2.5\) hours.)
A sample of three independent normal observations has unknown mean and variance \(\sigma^2>0\). The observed values are \(1,\ 3,\ 5\). Find both joint MLEs. (Answer: \(\hat{\mu}=3\) and \(\hat{\sigma}^2=8/3\).)
Four independent observations are modeled as normal with unknown \(\mu\) and \(\sigma^2>0\). All four observed values equal \(4\). Does a joint MLE exist? (Answer: No; at \(\mu=4\), the likelihood is unbounded as \(\sigma^2\downarrow0\).)
Independent observations follow a uniform distribution on \([0,\theta]\), where \(\theta>0\). The observed values are \(1,\ 2,\ 2,\ 5\). Find the MLE of \(\theta\) and state why it is a boundary maximum. (Answer: \(\hat{\theta}=5\), the smallest feasible endpoint parameter.)
Four independent observations follow a uniform distribution on an unknown interval \([\alpha,\beta]\), with \(\alpha<\beta\). The observed values are \(10,\ 12,\ 13,\ 15\). Find the endpoint MLEs and the maximum likelihood value. (Answer: \(\hat{\alpha}=10,\ \hat{\beta}=15\), and the maximum likelihood is \(5^{-4}=1/625\).)

What You'll Learn