Taking Logarithms Of Likelihoods

Why Take Logarithms?

For independent complete observations, the likelihood is a product: each observation contributes a density value or a probability mass, and the factors are multiplied. With many observations, that product can be difficult to simplify and can become extremely small numerically. Taking its logarithm changes the product into a sum, which is usually easier to differentiate, evaluate, and check.

The log-likelihood is not a different model or a different estimation method. It is a convenient transformation of the same likelihood. Because the logarithm is strictly increasing on positive values, any parameter value that maximizes a positive likelihood also maximizes its logarithm.

Definition: For complete observations \(x_1,\ldots,x_n\) with likelihood \(L(\theta)\), the log-likelihood is \(\ell(\theta)=\log L(\theta)\), where \(\log\) denotes the natural logarithm. For independent observations with densities \(f_i(x_i;\theta)\), \(L(\theta)=\prod_{i=1}^n f_i(x_i;\theta)\), so \(\ell(\theta)=\sum_{i=1}^n \log f_i(x_i;\theta)\). For discrete observations, replace each density by its probability mass function.

The observations are treated as fixed when writing a likelihood: the likelihood is a function of the unknown parameter \(\theta\). The product form requires independence. The observations need not have identical distributions, provided each contribution is written using the appropriate density or mass. A likelihood based on censored or truncated records must use the contribution for those records; simply applying a complete-data density to such records is not justified.

From a Product to a Sum

The key algebraic rule is \(\log(ab)=\log(a)+\log(b)\) for positive \(a\) and \(b\). Repeated application gives \(\log\left(\prod_i a_i\right)=\sum_i\log(a_i)\). In a parametric model, this often changes a product of terms involving the parameter into a sum whose derivatives are straightforward.

\ell(\theta)=\log L(\theta) =\log\left(\prod_{i=1}^n f_i(x_i;\theta)\right) =\sum_{i=1}^n \log f_i(x_i;\theta).

A density can exceed 1, so a density value is not necessarily a probability. Consequently, a likelihood or log-likelihood need not be bounded by 1 or by 0, respectively. The logarithm is defined when the likelihood is positive. If a parameter value assigns zero likelihood to the observed data, its log-likelihood is \(-\infty\), interpreted as a value that cannot maximize the likelihood when some other parameter gives positive likelihood.

Theorem: If \(L(\theta)>0\) over the parameter values being compared, then \(\widehat{\theta}\) maximizes \(L(\theta)\) if and only if it maximizes \(\ell(\theta)=\log L(\theta)\). This equivalence follows because \(\log\) is strictly increasing: \(L(\theta_1)>L(\theta_2)\) exactly when \(\log L(\theta_1)>\log L(\theta_2)\).

Worked Log-Likelihood Calculations

Worked Example: A Bernoulli Sample and Its Log-Likelihood

A hypothetical portfolio review records whether each of 10 independent, comparable claims exceeds a specified threshold. Let \(X_i=1\) indicate an exceedance and \(X_i=0\) otherwise. Assume \(X_i\) has a Bernoulli distribution with unknown exceedance probability \(p\), where \(0<p<1\). The observed sample contains 7 exceedances and 3 non-exceedances. Write the log-likelihood and find its maximizing value.

Step 1: Write the complete-data likelihood. For each observation, the probability mass is \(p^{X_i}(1-p)^{1-X_i}\). Independence gives \(L(p)=\prod_{i=1}^{10}p^{X_i}(1-p)^{1-X_i}=p^7(1-p)^3\).

Step 2: Take logarithms. \(\ell(p)=\log(p^7(1-p)^3)=7\log p+3\log(1-p)\). The product has become a sum, with one term for each observed outcome category.

Step 3: Differentiate and solve. \(\ell'(p)=7/p-3/(1-p)\). Setting this equal to zero gives \(7(1-p)=3p\), so \(7=10p\), and \(\widehat p=0.7\).

Step 4: Check the candidate. \(\ell''(p)=-7/p^2-3/(1-p)^2<0\) for \(0<p<1\). Thus the log-likelihood is strictly concave, and the stationary point is its maximum. At the estimate, \(\ell(0.7)=7\log(0.7)+3\log(0.3)\). Using \(\log(0.7)\approx-0.35667494\) and \(\log(0.3)\approx-1.20397280\), this is \(-2.49672458-3.61191840=-6.10864298\), or \(-6.1086\) to four decimal places.

Method and trap avoided. The method is to form the independent Bernoulli likelihood, take its log, and maximize the resulting sum. If only the count of exceedances is used, a binomial coefficient may appear in the count likelihood; it is constant in \(p\), so it does not change the maximizer. It should not be confused with a parameter-dependent term.

Worked Example: Exponential Observations and the Sample Size

Suppose three hypothetical independent claim-processing times, measured in hours, are observed as \(2\), \(5\), and \(8\). Assume each time has exponential density \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x\geq 0\), with unknown rate \(\lambda>0\) per hour. Find the log-likelihood and its maximizing rate.

Step 1: Form the likelihood. There are three density contributions, so \(L(\lambda)=(\lambda e^{-2\lambda})(\lambda e^{-5\lambda})(\lambda e^{-8\lambda})=\lambda^3e^{-15\lambda}\). The exponent contains the sum \(2+5+8=15\) hours.

Step 2: Take logarithms. \(\ell(\lambda)=3\log\lambda-15\lambda\). The coefficient 3 comes from the three factors of \(\lambda\); it is the sample size.

Step 3: Differentiate and solve. \(\ell'(\lambda)=3/\lambda-15\). The stationary point satisfies \(3/\lambda=15\), so \(\widehat\lambda=3/15=0.2\) per hour.

Step 4: Verify it is a maximum. \(\ell''(\lambda)=-3/\lambda^2<0\) for every \(\lambda>0\). The log-likelihood is strictly concave, so the stationary point maximizes it. As a numerical check, \(\ell(0.2)=3\log(0.2)-15(0.2)\approx3(-1.60943791)-3=-7.82831373\), or \(-7.8283\) to four decimal places.

Method and trap avoided. Differentiate the log-likelihood with the correct coefficient \(n=3\), then check concavity. Using \(1/\sum_i x_i\) would give \(1/15\), which incorrectly drops the three factors of \(\lambda\). The rate estimate is \(n/\sum_i x_i=3/15\), not \(1/\sum_i x_i\). Its unit is per hour, since the observed times are measured in hours.

Worked Example: A Normal Sample With Known Variance

Three hypothetical complete severity measurements, in thousands of dollars, are \(1\), \(3\), and \(4\). Assume they are independent observations from \(X_i\sim N(\mu,4)\), where the variance \(4\) is known and the mean \(\mu\) is unknown. Find the log-likelihood as a function of \(\mu\), and maximize it.

Step 1: Write the density contributions. The density for each observation is \(f(x_i;\mu)=\frac{1}{\sqrt{8\pi}}\exp[-(x_i-\mu)^2/8]\). The denominator 8 in the exponent is \(2\) times the known variance \(4\).

Step 2: Take the log of the product. For \(n=3\), \(\ell(\mu)=-\frac{3}{2}\log(8\pi)-\frac{1}{8}\sum_{i=1}^3(x_i-\mu)^2\). The first term is constant in \(\mu\), but retaining it gives the full log-likelihood.

Step 3: Differentiate. \(\ell'(\mu)=\frac{1}{4}\sum_{i=1}^3(x_i-\mu)\). Setting the derivative to zero yields \(1+3+4-3\mu=0\), hence \(\widehat\mu=8/3\), in thousands of dollars.

Step 4: Check the maximum and evaluate. \(\ell''(\mu)=-3/4<0\), so the stationary point is the unique maximum. The squared deviations at \(\mu=8/3\) are \((-5/3)^2=25/9\), \((1/3)^2=1/9\), and \((4/3)^2=16/9\). Their sum is \(42/9=14/3\). Thus \(\ell(8/3)=-\frac{3}{2}\log(8\pi)-\frac{1}{8}(14/3)\). Using \(\log(8\pi)\approx3.22417143\), this is approximately \(-4.83625714-0.58333333=-5.41959047\), or \(-5.4196\) to four decimal places.

Method and trap avoided. The method is to sum the three log densities and maximize the result. The constant term can be omitted when finding the maximizer, but dropping the squared-deviation term or using the variance as the standard deviation would change the model and the result.

Constants, Derivatives, and Numerical Checks

A log-likelihood often contains terms that depend on the observations but not on the parameter. For example, in the normal example, \(-\frac{3}{2}\log(8\pi)\) is constant in \(\mu\). Such a term may be omitted when differentiating or comparing candidate values of \(\mu\), because adding the same constant to every value does not change their ordering. Keep it when the actual log-likelihood value is requested.

Likewise, a likelihood may include a multiplicative factor independent of the parameter. Taking logs changes that factor into an additive constant. It is safe to omit that constant for locating the maximum, but not when reporting the full log-likelihood or comparing models in a calculation that requires the full likelihood values.

Justification: Suppose \(\ell(\theta)=g(\theta)+C\), where \(C\) does not depend on \(\theta\). Then \(\ell'(\theta)=g'(\theta)\) wherever the derivatives exist, and \(\ell(\theta_1)-\ell(\theta_2)=g(\theta_1)-g(\theta_2)\). Thus the constant changes neither stationary points nor comparisons between parameter values. This is why parameter-free terms may be dropped during maximization, but must be restored for a full log-likelihood calculation.

A derivative equal to zero identifies an interior stationary point, not automatically a maximum. Check the second derivative when available, or compare the log-likelihood over the allowed parameter values. A maximum may occur at a boundary, or a stationary point may be a minimum. The key practical benefit of the log form is that sums and derivatives are easier to inspect; it does not remove the need to check the result.

Common Exam Traps

Dropping the sample-size coefficient. For \(n\) exponential observations, the log-likelihood contains \(n\log\lambda\), not just \(\log\lambda\). Its derivative is \(n/\lambda-\sum_i x_i\).
Logging each term incorrectly. \(\log(a+b)\) is not \(\log a+\log b\). The product-to-sum rule applies to products, as in \(\log(ab)=\log a+\log b\).
Omitting terms without checking dependence. A term can be ignored in maximization only if it is constant in the parameter being estimated.
Confusing likelihood and probability. A likelihood is a function of the parameter for the observed data; for continuous data, density values are not probabilities of exact observations.
Ignoring units. In the exponential model, \(\lambda\) is a rate. If times are recorded in hours, the estimated rate is per hour; changing the time unit changes the numerical rate.

Key takeaway: For independent complete data, take the natural logarithm of each density or mass contribution and add the results. The log transformation preserves the maximizing parameter, exposes useful derivatives, and makes numerical checks easier, provided parameter-dependent terms and sample-size factors are retained.

Check Your Understanding

For each problem, write or use the log-likelihood before checking the answer.

Four independent Bernoulli observations contain 3 successes. If the success probability is \(p\), give the log-likelihood and its maximizing value. (Answer: \(\ell(p)=3\log p+\log(1-p)\); \(\widehat p=0.75\).)
Two independent exponential observations are \(4\) and \(6\) hours, with density \(f(x;\lambda)=\lambda e^{-\lambda x}\). Find the log-likelihood and maximizing rate, including its unit. (Answer: \(\ell(\lambda)=2\log\lambda-10\lambda\); \(\widehat\lambda=0.2\) per hour.)
For the exponential sample \(1,2,4\), calculate \(\ell(0.5)-\ell(0.4)\). Give the difference to four decimal places. (Answer: \(3\log(1.25)-7(0.1)\approx-0.0306\).)
Three independent observations are \(2,4,6\) from \(N(\mu,9)\), with known variance 9. Give the log-likelihood up to an additive constant and find its maximizing \(\mu\). (Answer: \(\ell(\mu)=-\frac{1}{18}\sum_{i=1}^3(x_i-\mu)^2+C\); \(\widehat\mu=4\).)
A complete sample from an exponential rate model contains \(n=5\) observations whose sum is 40 days. Find the maximizing rate and explain why \(1/40\) is not the answer. (Answer: \(\widehat\lambda=5/40=0.125\) per day; the likelihood has five factors of \(\lambda\), so its log contains \(5\log\lambda\).)

What You'll Learn