Why Likelihood and Probability Are Easy to Confuse
The same model formula can answer two different questions. A probability calculation fixes the parameter and asks how likely some possible data are. A likelihood calculation fixes the data that were observed and compares how well different parameter values account for them. The numerical expression may be the same, but which quantity varies is different.
This distinction matters throughout maximum likelihood estimation. It is especially easy to miss in a discrete model, where a likelihood at a parameter value is also the probability of the observed outcome at that value. In a continuous model, the distinction is sharper: the likelihood is based on a density, and a density value is not the probability of observing the exact data point.
For a discrete observation, \(L(\theta;x)\) is the probability of that exact observed outcome under \(\theta\). For a continuous observation, it is the density at the observed value, or the joint density at the observed sample. In neither case does a likelihood value, by itself, give the probability that \(\theta\) is true. A likelihood is not generally a probability distribution over parameter values: its values need not sum or integrate to one over \(\theta\).
One Formula, Two Questions
Suppose \(X\) has probability mass function \(p_\theta(x)\), or density \(f_\theta(x)\). The probability question fixes \(\theta\) and varies the possible data \(x\). The likelihood question fixes the observed value \(x\) and varies \(\theta\):
For a sample \(X_1,\ldots,X_n\), if the observations are independent under the model, the joint probability mass or density is the product of the individual mass functions or densities. Evaluating that product at the observed data gives the likelihood. Independence is an assumption for using this product; without independence, use the appropriate joint mass function or density instead.
A likelihood ratio compares two parameter values for the same observed data. If \(\theta_1\) and \(\theta_0\) are two candidate values, the ratio \(L(\theta_1;x)/L(\theta_0;x)\) describes their relative likelihood for those data. It is not, by itself, a probability ratio or a posterior odds calculation.
Worked Examples
Worked Example: A Binomial Likelihood Is Also an Outcome Probability
A hypothetical quality audit independently checks \(n=5\) circuit modules. Each module has probability \(p\) of passing the test, and \(K\) is the number that pass. The audit observes \(K=3\). Calculate the likelihood at \(p=0.60\) and \(p=0.50\), and explain what the first value means.
Method. Under the binomial model, \(K\sim\operatorname{Binomial}(5,p)\). For the observed count \(k=3\),
At \(p=0.60\), \(\binom{5}{3}=10\), \(0.60^3=0.216\), and \(0.40^2=0.16\). Therefore,
At \(p=0.50\), \(0.50^3=0.125\) and \(0.50^2=0.25\), so
For the fixed observation \(K=3\), \(p=0.60\) gives a larger likelihood than \(p=0.50\). The value \(0.3456\) is also the probability of observing exactly three passes if \(p=0.60\) is fixed. It is not the probability that \(p=0.60\) is true. As a check, the ratio is \(0.3456/0.3125=1.10592\), so the observed count is about 1.1059 times as likely under \(p=0.60\) as under \(p=0.50\). This comparison avoids the common mistake of treating an outcome probability as a probability assigned to the parameter.
Worked Example: A Normal Density and a Probability Interval
Suppose a hypothetical instrument reading \(X\), measured in millimetres, follows \(N(\theta,4)\), where the variance is known to be \(4\) and \(\theta\) is the unknown mean. One reading is \(x=10\). Calculate the likelihood at \(\theta=10\) and \(\theta=9\), then find the probability that a reading falls between 9 and 11 when \(\theta=10\).
Method. The standard deviation is \(\sqrt{4}=2\). The normal density evaluated at \(x=10\) is
At \(\theta=10\), the exponent is zero, giving
At \(\theta=9\), the squared difference is \((10-9)^2=1\), so
The ratio can be checked directly from the exact expressions: \(L(10;10)/L(9;10)=e^{1/8}\approx1.1331\). The observed value has greater density under \(\theta=10\). These are density values, not probabilities of the exact reading \(X=10\).
For the interval probability at \(\theta=10\), standardize both endpoints using mean 10 and standard deviation 2. The standardized endpoints are \((9-10)/2=-0.5\) and \((11-10)/2=0.5\). Thus,
The interval probability is about \(0.3829\) to four decimal places. If the CDF values are separately displayed rounded to \(0.6915\) and \(0.3085\), their displayed subtraction is \(0.3830\); the more accurate CDF difference rounds to \(0.3829\). The distinction is important: the density at one point and the probability over an interval are different quantities. The correction avoids both calling a density a point probability and creating a rounding inconsistency.
Worked Example: A Continuous Likelihood Can Exceed One
A hypothetical sensor records two independent waiting times, \(x_1=0.1\) hour and \(x_2=0.2\) hour. Each waiting time has an exponential distribution with rate \(\lambda\) per hour, so its density is \(f_\lambda(x)=\lambda e^{-\lambda x}\) for \(x\geq0\). Calculate the joint likelihood at \(\lambda=2\) and \(\lambda=1\), and compare it with the probability of the event \(0.1\leq X_1<0.2,\ 0.2\leq X_2<0.3\) when \(\lambda=2\).
Method. Independence gives the joint density, evaluated at the two observed times:
At the rate \(\lambda=2\) per hour,
At \(\lambda=1\) per hour,
The likelihood ratio is \(L(2)/L(1)=4e^{-0.3}\approx2.9633\). The value \(2.195247\) is greater than one, which is permitted because it is a density, with units of inverse hours squared, not a probability.
Now calculate the requested event probability. For an exponential waiting time, \(P(a\leq X<b)=e^{-\lambda a}-e^{-\lambda b}\) when \(0\leq a<b\). At \(\lambda=2\), the first interval has probability \(e^{-0.2}-e^{-0.4}\approx0.148411\), and the second has probability \(e^{-0.4}-e^{-0.6}\approx0.121508\). Independence gives
The event probability is between zero and one; the likelihood density is not required to be. This example avoids the trap of rejecting a valid continuous likelihood merely because its numerical value exceeds one.
Why the Likelihood Formula Holds
The definition follows from the model for the data. For a discrete sample, the joint mass function gives the probability of each possible sample. Once a sample \(x\) has been observed, evaluating that mass at \(x\) gives the probability assigned to that observed sample under each candidate parameter. Holding \(x\) fixed and allowing the candidate parameter to vary produces \(L(\theta;x)\).
For continuous data, a density describes probability per unit of measurement, rather than probability at an exact point. For a small interval of width \(\Delta x\) around \(x\), the model gives approximately \(f_\theta(x)\Delta x\) probability, when the density is suitably well-behaved near \(x\). The factor \(\Delta x\) does not depend on \(\theta\), so the density at the observed value describes the relative support of candidate parameters for that local observation. This is why the density, rather than a probability of an exact point, is used in the likelihood.
For independent observations, multiplication follows from the product rule for their joint mass or density. For example, with independent continuous observations \(x_1,\ldots,x_n\), the joint density is \(\prod_{j=1}^n f_\theta(x_j)\); hence the likelihood is that product with each \(x_j\) fixed. The same reasoning applies to independent discrete observations using their probability masses.
For a probability question, fix the parameter and identify an event in the data. For a likelihood question, fix the observed data.
Use a probability mass for discrete data, a density for continuous data, or the joint mass or density for a sample.
An event probability is between zero and one. A density-based likelihood need not be, and neither type of likelihood value is the probability that a parameter is true.
Common Exam Traps and Practical Interpretation
In a discrete model, the equality between a likelihood value and an outcome probability can obscure the change in viewpoint. For instance, \(L(0.60;3)=0.3456\) in the binomial example is indeed \(P(K=3\mid p=0.60)\). The incorrect interpretation is “the probability that \(p=0.60\) is 0.3456.” The calculation conditions on a specified \(p\); it does not calculate a probability for \(p\).
In a continuous model, the common error is to write \(P(X=x)=f_\theta(x)\). For a continuous random variable, \(P(X=x)=0\) at any single point, while \(f_\theta(x)\) can be positive and can exceed one. To find a probability, integrate the density over an event, or use an appropriate CDF difference. To evaluate a continuous likelihood at an observed value, use the density there.
Another useful safeguard is to label every numerical answer: “probability of the event,” “density at the observation,” or “likelihood at the candidate parameter.” Likelihood comparisons are meaningful when made for the same observed data and model. The numerical size of a continuous density can also depend on the measurement units: changing from hours to minutes changes density values by a scale factor, even though probabilities for corresponding physical intervals are unchanged.
These distinctions support the maximum likelihood methods aligned with the SOA VEE Mathematical Statistics syllabus. The central question is always whether the calculation varies over possible data or over parameter values. Stating that choice before calculating prevents a correct formula from being given an incorrect interpretation.
Check Your Understanding
For each question, identify whether the requested quantity is a probability or a likelihood and show the relevant calculation.
- A hypothetical batch has \(K\sim\operatorname{Binomial}(4,p)\), and an inspection observes \(K=1\). Calculate \(L(0.25;1)\). State what probability this number equals and what it does not mean. (Answer: \(L(0.25;1)=4(0.25)(0.75)^3=0.421875\); it equals \(P(K=1\mid p=0.25)\), not the probability that \(p=0.25\).)
- Let \(X\sim N(\theta,1)\), with variance \(1\), and suppose \(x=2\) is observed. Find \(L(2;2)\), \(L(1;2)\), and their ratio. Explain why \(L(2;2)\) is not \(P(X=2\mid\theta=2)\). (Answer: \(L(2;2)=1/\sqrt{2\pi}\approx0.3989\), \(L(1;2)=e^{-1/2}/\sqrt{2\pi}\approx0.2420\), and the ratio is \(e^{1/2}\approx1.6487\); the exact-point probability is zero.)
- Three independent hypothetical counts have Poisson rates \(\lambda\) and observed values \(0,1,2\). Compare the joint likelihood at \(\lambda=1\) with that at \(\lambda=2\). (Answer: \(L(\lambda)=\lambda^3e^{-3\lambda}/2\), so \(L(1)=e^{-3}/2\approx0.0249\), \(L(2)=4e^{-6}\approx0.0099\), and \(L(1)/L(2)=e^3/8\approx2.5107\).)
- A hypothetical waiting time has an exponential distribution with rate \(\lambda=2\) per hour. Find the density at \(x=0.5\) hour and the probability that the waiting time is between \(0.4\) and \(0.6\) hour. (Answer: \(f_2(0.5)=2e^{-1}\approx0.7358\), while \(P(0.4\leq X\leq0.6)=e^{-0.8}-e^{-1.2}\approx0.1481\).)
- Suppose \(X\sim N(\theta,9)\), with variance \(9\), and the observed value is \(x=4\). Compare the likelihood at \(\theta=4\) with that at \(\theta=1\), and find \(P(1\leq X\leq7\mid\theta=4)\). (Answer: \(L(4;4)/L(1;4)=e^{1/2}\approx1.6487\), and the interval probability is \(\Phi(1)-\Phi(-1)\approx0.6827\).)