Why Work With Log-Likelihoods?
For independent observations, the likelihood is a product of contributions from the individual observations. Products can be awkward to simplify, compare, and differentiate. Taking a logarithm turns that product into a sum, while preserving which parameter value maximizes it. This makes the log-likelihood a convenient representation of the same evidence in the data, not a different estimation method.
A second simplification is often possible: a positive factor that depends on the observed data but not on the parameter can be omitted when maximizing. The condition matters. In particular, a term that restricts which parameter values are possible is not a parameter-free factor, even if it is written beside other terms in a likelihood.
For a discrete model, the likelihood is based on the probability of the observed outcomes. For a continuous model, it is based on the joint density evaluated at the observed values. In either case, the parameter space and any support restrictions belong to the model and must be respected.
Products Become Sums
Suppose \(X_1,\ldots,X_n\) are independent observations from a model with probability mass function or density \(f(x;\theta)\). For observed values \(x_1,\ldots,x_n\), independence gives the likelihood as a product. Wherever all factors are positive, the logarithm of that product is the sum of their logarithms.
This identity is a direct consequence of the logarithm rule \(\log(ab)=\log a+\log b\), applied repeatedly. If any likelihood factor is zero, the joint likelihood is zero and its log is conventionally \(-\infty\). Thus the log form must preserve zero-likelihood parameter values rather than treating them as ordinary finite terms.
For example, a likelihood may contain a product of powers such as \(\theta^r(1-\theta)^s\). Its log is \(r\log\theta+s\log(1-\theta)\), provided the parameter values make the original likelihood positive. The log form is usually easier to manipulate, but it does not remove restrictions such as \(0<\theta<1\).
When a Factor Can Be Removed
Suppose a likelihood can be written as \(L(\theta;x)=c(x)g(\theta;x)\), where \(c(x)>0\) is fixed once the observed data are fixed. For every admissible \(\theta\), multiplication by this same positive number preserves the ordering of likelihood values. Thus \(L\) and \(g\) have exactly the same maximizing parameter values. In log form, the same fact appears as an additive constant \(\log c(x)\), which does not vary with \(\theta\).
The word positive is essential: multiplying by zero would erase all distinctions between parameter values. The requirement that \(c(x)\) be independent of \(\theta\) is equally essential. A term that depends on the parameter can change the relative likelihoods and cannot generally be omitted.
| Likelihood component | Can it be removed for maximization? | Reason |
|---|---|---|
| A positive numerical factor determined only by the observed data | Yes | It multiplies every admissible likelihood value by the same positive number. |
| A term involving the parameter, such as \(\theta^r\) | No, not in general | Its value changes with \(\theta\) and may change the likelihood ordering. |
| An indicator restricting which parameter values permit the observations | No | It can make the likelihood zero for some parameter values and positive for others. |
Worked Example: Ordered Bernoulli Outcomes Versus a Success Count
A hypothetical device produces an alert on each independent inspection with probability \(\theta\), where \(0<\theta<1\). Five inspections produce the ordered outcomes alert, no alert, alert, alert, no alert. Find the likelihood and log-likelihood for the recorded sequence. Then compare the likelihoods at \(\theta=0.4\) and \(\theta=0.6\), both for the sequence and for the observation that there were three alerts in five inspections.
Method. Use independence to multiply the probabilities of the observed outcomes, then take natural logarithms. For the count observation, include the number of distinct ordered sequences with three alerts.
The ordered sequence contains three alerts and two non-alerts, so its likelihood is
If only the count of alerts is recorded, there are \(\binom{5}{3}=10\) possible sequences with three alerts. The count likelihood is therefore
At \(\theta=0.4\), the sequence likelihood is \(0.4^3(0.6)^2=0.064(0.36)=0.02304\). The count likelihood is \(10(0.02304)=0.2304\). Its log-likelihood is
At \(\theta=0.6\), the sequence likelihood is \(0.6^3(0.4)^2=0.216(0.16)=0.03456\). The count likelihood is \(10(0.03456)=0.3456\), and
The count likelihood is the sequence likelihood multiplied by the positive, parameter-free factor \(10\). Both representations rank \(\theta=0.6\) above \(\theta=0.4\), as do their log-likelihoods. The factor changes the numerical likelihood and adds \(\log 10\) to the log-likelihood, but it does not change which parameter values maximize it. Common mistake avoided: omitting the factor \(10\) when writing the probability of the count. It is appropriate to omit that factor only when the goal is maximizing over \(\theta\), not when reporting the count's probability or likelihood value.
Worked Examples With Continuous Likelihoods
Worked Example: A Normal Mean With Known Variance
A hypothetical laboratory measures three independent calibration errors, in fixed measurement units: \(8\), \(10\), and \(12\). Assume each observation has distribution \(N(\mu,4)\), where \(\mu\) is unknown and the variance \(4\) is known. Write the likelihood and log-likelihood. Identify a positive parameter-free factor, and compare the likelihoods at \(\mu=10\) and \(\mu=11\).
Method. Multiply the three normal densities and use the observed values to separate parameter-free terms from terms that involve \(\mu\). The variance is \(4\), so the normal density has standard deviation \(2\).
For one observation \(x_i\), the density is \((2\pi\cdot4)^{-1/2}\exp[-(x_i-\mu)^2/(2\cdot4)]\). Multiplying the three densities gives
To simplify the squared deviations, write them around \(10\):
Consequently,
and the log-likelihood is
The factor \((8\pi)^{-3/2}e^{-1}\) is positive and does not depend on \(\mu\), so it can be omitted when comparing candidate means for maximization. At \(\mu=10\), the remaining factor is \(1\). At \(\mu=11\), it is \(\exp[-3(1)^2/8]=e^{-3/8}\approx0.6873\). Thus \(L(11)/L(10)=0.6873\), so the likelihood at \(11\) is lower. The exact log-likelihood difference is \(\ell(11)-\ell(10)=-3/8=-0.375\). These calculations agree because \(\exp(-0.375)\approx0.6873\). Common mistake avoided: treating the known-variance normalizing constant as parameter-dependent. It is fixed here because the variance is known; the model would need to be reconsidered if the variance were also unknown.
Worked Example: A Uniform Model With Parameter-Dependent Support
A hypothetical sensor records three independent positive waiting times, in minutes: \(1.2\), \(1.7\), and \(2.1\). Assume each has a uniform distribution on \((0,\theta)\), where \(\theta>0\) is unknown. Write the likelihood, and compare the likelihoods at \(\theta=2\) and \(\theta=3\).
Method. Use the uniform density together with its support condition. The support condition must remain in the likelihood because it depends on \(\theta\).
One observation has density \(1/\theta\) when \(0<x<\theta\) and density zero otherwise. The largest observed waiting time is \(2.1\), so all three density contributions are positive only when \(\theta>2.1\). For the observed data, the joint likelihood is
where the indicator equals \(1\) when its condition holds and \(0\) otherwise. At \(\theta=2\), the support condition fails, so \(L(2)=0\). At \(\theta=3\), it holds, so
If the indicator were wrongly discarded, the expression \(\theta^{-3}\) alone would give \(2^{-3}=0.125\) at \(\theta=2\) and \(3^{-3}\approx0.03704\) at \(\theta=3\). That incorrect expression would rank \(\theta=2\) higher, even though a model with upper endpoint \(2\) cannot produce the observed value \(2.1\). The indicator is not a positive parameter-free factor; it determines whether the likelihood is zero. Common mistake avoided: removing a support restriction as though it were a harmless constant.
Practical Checks and Common Traps
Before simplifying a likelihood, keep the parameter space visible and check what each term does. In particular, separate factors that affect numerical values from factors that determine whether a value is possible. This distinction is useful when the likelihood is written in a compact product form or when the data are summarized as counts.
Use the joint mass function or density under the stated sampling model, including any support restrictions.
Convert products into sums, and keep terms that involve the parameter.
A removable factor must be positive and independent of the parameter over the stated parameter space.
Parameter-free positive factors may change likelihood values, but they cannot change their ordering or maximizing values.
A frequent trap is to remove every term that appears not to contain the parameter at first glance. For example, the indicator in a parameter-dependent support model may look like a separate multiplier, but it can switch the likelihood from zero to positive. Another trap is to use a count likelihood but omit its combinatorial factor when asked for the probability of that count. The factor may be irrelevant to maximizing over the parameter while still being essential to the likelihood's numerical value.
Working with log-likelihoods and parameter-free factors is aligned with the SOA VEE Mathematical Statistics syllabus. The central discipline is to simplify without changing either the parameter space or the relative likelihood of its admissible values.
Check Your Understanding
For each problem, write the relevant likelihood or log-likelihood and identify which factors, if any, may be removed when maximizing over the parameter.
- Six independent inspections each produce a pass with probability \(p\). The recorded ordered outcomes contain four passes and two failures. Give the log-likelihood for this sequence. (Answer: \(4\log p+2\log(1-p)\), for \(0<p<1\).)
- Three independent counts from a Poisson distribution with mean \(\lambda>0\) are \(2,0,3\). Write the likelihood and log-likelihood, and identify the parameter-free factor. (Answer: \(L(\lambda)=e^{-3\lambda}\lambda^5/12\); \(\ell(\lambda)=-3\lambda+5\log\lambda-\log 12\); the factor \(1/12\) may be omitted for maximizing.)
- Two independent measurements are \(4\) and \(6\), each distributed as \(N(\mu,1)\) with known variance \(1\). Compare the likelihood at \(\mu=5\) with that at \(\mu=6\) by calculating the likelihood ratio \(L(6)/L(5)\). (Answer: \(e^{-1}\approx0.3679\).)
- Four independent observations from a uniform distribution on \((0,\theta)\) are \(0.5,1.4,2.0,2.6\). Give the likelihood at \(\theta=2.5\) and at \(\theta=3\), retaining the support condition. (Answer: \(L(2.5)=0\) and \(L(3)=3^{-4}=1/81\approx0.01235\).)
- A binomial count has \(n=8\) trials and the observed count is \(k=2\), with success probability \(p\). Write the count log-likelihood and state the difference between it and the log-likelihood for one particular ordered sequence with two successes. (Answer: \(\ell(p)=\log 28+2\log p+6\log(1-p)\); it exceeds the ordered-sequence log-likelihood by the parameter-free constant \(\log 28\).)