Why Transform an MLE?
A model is often written in terms of one parameter even though the quantity of interest is another. For example, a normal model may be parameterized by its variance, while a report requests the standard deviation. An exponential waiting-time model is commonly described by its rate, while an application may ask for the mean waiting time. The invariance property of maximum likelihood estimation lets us move from an MLE of the model parameter to an MLE of a transformed parameter.
The transformation is direct, but it does not remove the need to check that the original MLE exists. In particular, a normal sample with no variation and an exponential sample consisting entirely of zeros create boundary problems. The usual formulas apply only when the likelihood has a finite maximum within the stated parameter space.
Normal Variance and Standard Deviation
Suppose \(X_1,\ldots,X_n\) are independent and identically distributed as \(N(\mu,\sigma^2)\), where \(\sigma^2>0\). The standard deviation is \(\sigma=\sqrt{\sigma^2}\). The normal likelihood can be maximized with respect to \(\sigma^2\), and invariance then gives the MLE of \(\sigma\) by taking a square root.
- If \(\mu\) is known and at least one observation differs from \(\mu\), then \(\widehat{\sigma^2}_{\mathrm{MLE}}=\frac{1}{n}\sum_{i=1}^n(X_i-\mu)^2\) and \(\hat{\sigma}_{\mathrm{MLE}}=\sqrt{\frac{1}{n}\sum_{i=1}^n(X_i-\mu)^2}\).
- If both \(\mu\) and \(\sigma^2\) are unknown and the observations are not all equal, then \(\hat{\mu}=\bar{X}\), \(\widehat{\sigma^2}_{\mathrm{MLE}}=\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2\), and \(\hat{\sigma}_{\mathrm{MLE}}=\sqrt{\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2}\).
The divisor is \(n\) because these are maximum likelihood estimates, not the usual unbiased sample variance. The unbiased sample variance uses a correction for estimating the mean; maximum likelihood instead maximizes the likelihood itself. The following calculation shows how the likelihood leads to the \(n\) divisor.
Why the Normal MLE Has a Divisor of n
For the unknown-mean case, the joint likelihood under the stated independent normal model is \(L(\mu,\sigma^2)=(2\pi\sigma^2)^{-n/2}\exp[-\sum_{i=1}^n(X_i-\mu)^2/(2\sigma^2)]\). For any fixed \(\sigma^2>0\), maximizing this likelihood over \(\mu\) is equivalent to minimizing the sum of squared deviations. That sum is minimized at \(\bar{X}\). Substitute \(\bar{X}\) and write \(Q=\sum_{i=1}^n(X_i-\bar{X})^2\). The log-likelihood as a function of \(\sigma^2\) is
Differentiating with respect to \(\sigma^2\) gives
Setting this derivative equal to zero yields \(Q=n\sigma^2\), so the candidate is \(\widehat{\sigma^2}=Q/n\). When \(Q>0\), the log-likelihood increases before this candidate and decreases after it, giving the maximum. If \(Q=0\), all observations are identical: the likelihood is proportional to \((\sigma^2)^{-n/2}\) and grows without bound as \(\sigma^2\) approaches zero. Since zero is excluded from the parameter space, no MLE exists. With known \(\mu\), the same argument uses \(Q=\sum(X_i-\mu)^2\); it also requires \(Q>0\).
Worked Example: Estimating Both Normal Parameters
A hypothetical instrument records four independent readings, in the same units, modeled as \(N(\mu,\sigma^2)\) with both parameters unknown and \(\sigma^2>0\). The readings are \(18,19,20,23\). Find the MLEs of the mean, variance, and standard deviation. Use the invariance property for the standard deviation.
Method and mean estimate. The readings are not all equal, so the normal MLE formulas apply. First calculate \(\bar{X}=(18+19+20+23)/4=80/4=20\). Thus \(\hat{\mu}=20\).
Variance estimate. The deviations from 20 are \(-2,-1,0,3\), and their squares are \(4,1,0,9\). Therefore, \(Q=4+1+0+9=14\), and \(\widehat{\sigma^2}_{\mathrm{MLE}}=Q/n=14/4=3.5\) squared units.
Standard deviation estimate by invariance. Since \(g(\sigma^2)=\sqrt{\sigma^2}\), \(\hat{\sigma}_{\mathrm{MLE}}=\sqrt{3.5}=1.870828\ldots\), or \(1.8708\) units to four decimal places.
Check and common mistake avoided. The unrounded standard deviation squares to \((\sqrt{3.5})^2=3.5\), as required. The sample mean is 20, not 15. Also, dividing the squared deviations by \(n-1=3\) would produce an unbiased variance estimate, not the MLE requested here.
Worked Example: Normal Variance With a Known Mean
A hypothetical calibration process produces four independent readings modeled as \(N(12,\sigma^2)\), where the mean of 12 units is known and \(\sigma^2>0\) is unknown. The readings are \(10,11,14,15\). Estimate the variance and standard deviation by maximum likelihood.
Method and condition. Use the known-mean normal formula. At least one observation differs from 12—in fact, all four do—so the sum of squared deviations is positive and an interior MLE exists.
The deviations from the known mean are \(-2,-1,2,3\); their squares are \(4,1,4,9\). Hence \(\sum(X_i-12)^2=4+1+4+9=18\).
The variance MLE is \(18/4=4.5\) squared units. By invariance, the standard deviation MLE is \(\sqrt{4.5}=2.121320\ldots\), or \(2.1213\) units to four decimal places.
Check and common mistake avoided. Squaring the unrounded standard deviation gives 4.5. The divisor remains the sample size \(n=4\), even though the mean is known. Do not divide by \(n-1\); the likelihood calculation gives a divisor of \(n\).
Exponential Mean From the Rate MLE
Let \(X_1,\ldots,X_n\) be independent exponential waiting times with rate \(\lambda>0\), using the density \(f_X(x)=\lambda e^{-\lambda x}\) for \(x\geq 0\). The model mean is \(E[X]=1/\lambda\). For a sample with positive total waiting time, the rate MLE is \(n/\sum X_i=1/\bar{X}\). Invariance transforms it into an MLE of the mean waiting time.
To see the transformation explicitly, the joint likelihood for nonnegative observations is \(L(\lambda)=\lambda^n e^{-\lambda\sum X_i}\). When the sum is positive, differentiating the log-likelihood \(n\log\lambda-\lambda\sum X_i\) gives the score \(n/\lambda-\sum X_i\). Its zero occurs at \(\hat{\lambda}=n/\sum X_i\), and the log-likelihood is maximized there. The mean parameter is \(g(\lambda)=1/\lambda\), so invariance gives \(g(\hat{\lambda})=\sum X_i/n=\bar{X}\). If the sum is zero, the likelihood is \(\lambda^n\), which grows without bound as \(\lambda\) increases; no finite maximizing rate exists.
Worked Example: Converting an Exponential Rate to a Mean
A hypothetical monitoring system records four independent waiting times, in hours, until a specified alert occurs. Assume each waiting time follows an exponential distribution with rate \(\lambda>0\) alerts per hour. The observations are \(2,3,4,7\) hours. Estimate the rate and the mean waiting time by maximum likelihood.
Method and condition. The exponential rate MLE is \(n/\sum X_i\), and the mean estimate is obtained by applying \(g(\lambda)=1/\lambda\). The total time is positive, so the finite-MLE condition holds.
The total is \(2+3+4+7=16\) hours, with \(n=4\). Thus \(\hat{\lambda}=4/16=0.25\) per hour.
By invariance, \(\widehat{E[X]}_{\mathrm{MLE}}=1/0.25=4\) hours. Equivalently, the observed sample mean is \(16/4=4\) hours.
Check and common mistake avoided. The units are reciprocal: a rate of \(0.25\) per hour corresponds to a mean of 4 hours, not \(0.25\) hours. Multiplying the estimated rate and mean gives \(0.25\times4=1\), as expected.
Conditions and Common Exam Traps
An invariance calculation is only as valid as the MLE being transformed. Before taking a square root or reciprocal, check the sample and the parameter space. A formal expression such as \(Q/n=0\) does not become a valid normal variance MLE when the model requires \(\sigma^2>0\); likewise, \(n/\sum X_i\) is not a finite rate estimate when the denominator is zero.
- Using the wrong normal divisor. The normal variance MLE uses \(n\), whether the mean is known or estimated. The divisor \(n-1\) belongs to the unbiased sample variance, not this likelihood maximization.
- Ignoring a zero residual sum of squares. If the normal observations are all identical, the variance candidate is zero and is outside \(\sigma^2>0\). The likelihood is unbounded toward that boundary, so report that no MLE exists.
- Ignoring zero exponential observations. If all waiting times are zero, the rate likelihood increases without bound as \(\lambda\) increases. There is no finite rate MLE and therefore no finite mean estimate obtained by transforming it.
- Reversing rate and mean. For the exponential model used here, the mean is \(1/\lambda\). Transform the rate estimate with a reciprocal, and keep the rate and mean units consistent.
These transformations are useful whenever the model parameter is not the quantity an application needs to report. For normal data, the variance and standard deviation describe spread in different units; for exponential data, the rate and mean express frequency and waiting time, respectively. In each case, invariance supplies the MLE for the transformed quantity, while the original likelihood conditions determine whether that estimate is actually defined.
Check Your Understanding
For each problem, state the relevant MLE and apply the transformation only if the required finite-MLE condition holds.
- Independent observations \(5,7,8\) follow \(N(\mu,\sigma^2)\) with both parameters unknown and \(\sigma^2>0\). Calculate \(\hat{\mu}\), the MLE of \(\sigma^2\), and the MLE of \(\sigma\). (Answer: \(\hat{\mu}=20/3\); \(\widehat{\sigma^2}_{\mathrm{MLE}}=14/9\); \(\hat{\sigma}_{\mathrm{MLE}}=\sqrt{14}/3\approx1.2472\).)
- Independent readings \(4,7,9\) follow \(N(6,\sigma^2)\), where the mean 6 is known. Calculate the variance and standard deviation MLEs. (Answer: \(\widehat{\sigma^2}_{\mathrm{MLE}}=14/3\); \(\hat{\sigma}_{\mathrm{MLE}}=\sqrt{14/3}\approx2.1602\).)
- Independent waiting times \(1,2,2,5\) have an exponential distribution with rate \(\lambda>0\), measured per time unit. Find the rate MLE and use invariance to estimate the mean waiting time. (Answer: \(\hat{\lambda}=4/10=0.4\) per time unit; \(\widehat{E[X]}_{\mathrm{MLE}}=2.5\) time units.)
- Three independent observations from \(N(\mu,\sigma^2)\) are \(7,7,7\), with both \(\mu\) and \(\sigma^2>0\) unknown. Calculate the usual variance candidate and decide whether an MLE exists in the stated parameter space. (Answer: the candidate is \(0\), but there is no MLE for \(\sigma^2>0\), because the likelihood is unbounded as \(\sigma^2\) approaches zero.)
- Four independent exponential waiting times with rate \(\lambda>0\) are all recorded as zero. Evaluate the rate likelihood and determine whether a finite rate MLE, and hence an MLE of the mean by rate transformation, exists. (Answer: \(L(\lambda)=\lambda^4\), which is unbounded as \(\lambda\to\infty\); there is no finite rate MLE and no transformed finite mean MLE.)