Likelihood Functions From Observed Data

What a Likelihood Describes

A probability model describes how possible observations behave when its parameter is specified. Once an observation has been recorded, we can turn that relationship around: hold the observed value fixed and examine how the model's probability or density for that value varies with the parameter. This function is the likelihood. It is a basic building block for maximum likelihood estimation, but defining and constructing it comes before trying to find a maximizing parameter.

Definition: Suppose $X$ has a probability mass function $p_X(x;\theta)$ in a discrete model, or a density $f_X(x;\theta)$ in a continuous model, where $\theta$ is an unknown parameter. After observing $X=x$, the likelihood is the function of $\theta$ $$ L(\theta;x)=p_X(x;\theta)\quad\text{(discrete)},\qquad L(\theta;x)=f_X(x;\theta)\quad\text{(continuous)}. $$ The observed value $x$ is held fixed; the parameter $\theta$ varies over its allowed parameter space.

The semicolon in $p_X(x;\theta)$ or $f_X(x;\theta)$ separates the observation argument from the parameter. A likelihood is not a probability distribution for $\theta$. In particular, its values need not sum or integrate to one as $\theta$ varies. A likelihood evaluates how well each parameter value's model accounts for the data under the specified probability model.

Constructing a Likelihood From a Discrete Model

For a discrete random variable, the probability mass function gives the probability of each possible outcome. To construct its likelihood, substitute the observed value for the outcome argument and leave the parameter variable. Keep the parameter space and the model's support visible: if the observation is impossible under a parameter value, the likelihood there is zero.

Worked Example: A Recorded Category

A hypothetical inspection records one component as either category A, B, or C. Under the model, the category probabilities are $P(A)=\theta$, $P(B)=1/2$, and $P(C)=1/2-\theta$, with $0\leq\theta\leq1/2$. The recorded category is C. Construct its likelihood, check the model, and evaluate the likelihood at $\theta=0.1$ and $\theta=0.4$.

Method: substitute the fixed observed category into its probability mass. First check that the probabilities are valid throughout the stated parameter space: $$ \theta+\frac12+\left(\frac12-\theta\right)=1. $$ When $0\leq\theta\leq1/2$, all three probabilities are nonnegative.

For the observed category C, the likelihood is $$ L(\theta;C)=P(X=C;\theta)=\frac12-\theta,\qquad 0\leq\theta\leq\frac12. $$ At $\theta=0.1$, $L(0.1;C)=0.5-0.1=0.4$. At $\theta=0.4$, $L(0.4;C)=0.5-0.4=0.1$. These are the model probabilities of the recorded category at those two parameter values. The likelihood is larger at $0.1$ than at $0.4$, but these evaluations alone do not determine any estimate.

Check: both parameter values lie in the allowed interval, and the likelihood values equal the corresponding category-C probabilities. Common mistake avoided: using $P(A)=\theta$ because it contains the parameter, even though the recorded category was C.

A likelihood may also have a piecewise form when the observation belongs to the model's support only for some parameter values. The zero part is not optional: it records that the model assigns no probability to the observed outcome under those parameter values.

Worked Example: A Discrete Uniform Endpoint

A discrete random variable is uniform on the integers $\{0,1,\ldots,m\}$, where $m$ is an unknown nonnegative integer. One value $X=3$ is observed. Construct the likelihood for $m$ and evaluate it at $m=2$, $m=3$, and $m=5$.

Method: use the discrete uniform mass and enforce its support. For a fixed nonnegative integer $m$, each of the $m+1$ possible values has probability $1/(m+1)$. The observation 3 is possible only if $m\geq3$. Therefore, $$ L(m;3)= \begin{cases} 0, & m<3,\\[4pt] \dfrac{1}{m+1}, & m\geq3, \end{cases} \qquad m\in\{0,1,2,\ldots\}. $$

At $m=2$, the support is $\{0,1,2\}$, so 3 is impossible and $L(2;3)=0$. At $m=3$, $L(3;3)=1/(3+1)=1/4$. At $m=5$, $L(5;3)=1/(5+1)=1/6$.

Check: when $m=3$, there are four equally likely outcomes, so the probability of 3 is $1/4$. When $m=5$, there are six, so it is $1/6$. Common mistake avoided: applying $1/(m+1)$ at $m=2$ without checking whether the observed value belongs to the support.

Constructing a Likelihood From a Density

For a continuous random variable, a density value at a point is not the probability of observing exactly that point. In fact, under a continuous model, $P(X=x)=0$ for each individual value $x$. The likelihood is still defined by evaluating the density at the observed value and treating that result as a function of the parameter. Thus a likelihood value from a density is not itself the probability of the observation.

Key distinction: In a continuous model, $L(\theta;x)=f_X(x;\theta)$, not $P(X=x;\theta)$. Probabilities are obtained by integrating the density over intervals. The density at an observed point supplies the likelihood function; it is not a point probability.

Worked Example: An Exponential Waiting Time

A hypothetical sensor's waiting time $X$, measured in years, is modeled by an exponential distribution with unknown rate $\theta>0$ per year. Its density is $f_X(x;\theta)=\theta e^{-\theta x}$ for $x\geq0$, and zero otherwise. One waiting time of $2.5$ years is observed. Construct the likelihood and evaluate it at $\theta=0.2$ and $\theta=0.4$ per year.

Method: substitute the observed time into the density. Since $2.5\geq0$, the observation is in the support for every allowed $\theta>0$. Therefore, $$ L(\theta;2.5)=f_X(2.5;\theta)=\theta e^{-2.5\theta},\qquad\theta>0. $$

At $\theta=0.2$ per year, $$ L(0.2;2.5)=0.2e^{-2.5(0.2)}=0.2e^{-0.5} =0.2(0.606531)\approx0.121306. $$ At $\theta=0.4$ per year, $$ L(0.4;2.5)=0.4e^{-2.5(0.4)}=0.4e^{-1} =0.4(0.367879)\approx0.147152. $$ The likelihood values are densities, with units of inverse years, rather than probabilities. The calculations use the rate in inverse years and the waiting time in years, so the exponent is dimensionless.

Check: substituting each rate into the stated density gives the displayed value; the larger density here is at $0.4$ per year. Common mistake avoided: calling $0.121306$ the probability that the waiting time equals exactly $2.5$ years. That probability is zero; the displayed value is a density.

Why the Density Gives the Likelihood

The link between a density and likelihood can be understood by considering a narrow interval around the observation. Suppose $X$ has a density $f_X(x;\theta)$ that is continuous in $x$ near the observed value $x$. For a small positive width $h$, the probability of observing a value in the interval from $x$ to $x+h$ is the integral of the density over that interval. The average probability per unit width approaches the density at $x$:

\frac{P(x\leq X\leq x+h;\theta)}{h} =\frac{1}{h}\int_x^{x+h} f_X(t;\theta)\,dt \longrightarrow f_X(x;\theta)\quad\text{as }h\to0^+.

The integral formula for interval probability is the defining relationship between a density and probabilities. Continuity ensures that, over a sufficiently short interval, the density is close to its value at $x$; consequently, the average density over that interval approaches $f_X(x;\theta)$. This explains why a density evaluated at the observed value is the continuous-model counterpart of a discrete probability mass. It does not turn the density into a probability at a point.

A Second Continuous Example: Normal Density

The same construction applies to other continuous models. The model must be stated clearly, including any parameters that are treated as known, before the density is evaluated.

Worked Example: A Normal Observation With Known Variance

A hypothetical measurement $X$, in centimeters, follows a normal distribution with unknown mean $\theta$ centimeters and known variance $0.25$ square centimeters. One measurement, $x=1.8$ centimeters, is recorded. Construct the likelihood and evaluate it at $\theta=1.3$ and $\theta=2.3$ centimeters.

Method: evaluate the specified normal density at the fixed observation. The standard deviation is $\sqrt{0.25}=0.5$ centimeters. The density is $$ f_X(x;\theta)=\frac{1}{\sqrt{2\pi(0.25)}}\exp\left(-\frac{(x-\theta)^2}{2(0.25)}\right). $$ Thus the likelihood is $$ L(\theta;1.8)=\frac{1}{\sqrt{0.5\pi}}\exp\left(-\frac{(1.8-\theta)^2}{0.5}\right). $$

At $\theta=1.3$, the difference $1.8-1.3=0.5$, so the exponent is $-(0.5)^2/0.5=-0.5$. Since $1/\sqrt{0.5\pi}\approx0.797885$ and $e^{-0.5}\approx0.606531$, $$ L(1.3;1.8)\approx0.797885(0.606531)\approx0.4840. $$ At $\theta=2.3$, the difference $1.8-2.3=-0.5$, whose square is also $0.25$. Hence $L(2.3;1.8)\approx0.4840$ as well.

Check: the two parameter values are equally far from the observation, and the normal density depends on their squared difference, so the likelihood values agree. Common mistake avoided: using $0.5$ as the variance. It is the standard deviation; the model specifies variance $0.25$.

Reading and Using Likelihoods Correctly

To construct a likelihood reliably, separate the roles of the observation and the parameter. First write the probability mass function or density for the stated model. Then substitute the observed data into the outcome position, retain the parameter as the variable, and impose the allowed parameter space and support. Only after these steps should you evaluate the function at specified parameter values.

State the model.
Identify whether the observation is discrete or continuous, give the relevant mass function or density, and state the parameter space.

Fix the observed data.
Replace the outcome argument by the recorded value. Do not replace the unknown parameter by a number unless evaluating at a specified parameter value.

Check support.
If the observation is impossible under a parameter value, the likelihood there is zero.

Interpret the result.
A discrete likelihood value is a probability mass. A continuous likelihood value is a density. Neither is a probability distribution over the parameter.

A frequent exam trap is to confuse the likelihood function with the probability of the data as a function of the data. The probability model is originally written with the parameter fixed and the possible outcome varying. The likelihood uses the same mass or density formula, but after observing the outcome, it treats that outcome as fixed and varies the parameter. The expression may look unchanged; the role of its arguments has changed.

Key takeaway: A likelihood is the model's probability mass or density evaluated at fixed observed data and viewed as a function of the unknown parameter. Always preserve the model's support, distinguish a density from a point probability, and do not interpret likelihood values as probabilities about the parameter.

Check Your Understanding

For each problem, construct or evaluate the likelihood while keeping the observation fixed.

A single device result is either Pass or Fail, with $P(\text{Pass})=\theta$ and $P(\text{Fail})=1-\theta$, where $0\leq\theta\leq1$. The result is Fail. Give the likelihood and its value at $\theta=0.25$. (Answer: $L(\theta;\text{Fail})=1-\theta$; $L(0.25;\text{Fail})=0.75$.)
$X$ is uniform on the integers $\{1,\ldots,k\}$, where $k$ is a positive integer. The observed value is 4. Give the likelihood at $k=3$ and $k=8$. (Answer: $L(3;4)=0$, $L(8;4)=1/8=0.125$.)
$X$ has a uniform density on $[0,\theta]$, with $\theta>0$. The observation is $x=3$. Give the likelihood as a piecewise function and evaluate it at $\theta=2$ and $\theta=6$. (Answer: $L(\theta;3)=0$ for $\theta<3$, and $1/\theta$ for $\theta\geq3$; the values are $0$ and $1/6$.)
A waiting time in years has exponential density $f_X(x;\theta)=\theta e^{-\theta x}$ for $x\geq0$, with rate $\theta>0$ per year. For the observation $x=1.2$ years, evaluate the likelihood at $\theta=0.5$ per year. (Answer: $0.5e^{-0.6}\approx0.2744$ per year.)
$X\sim N(\theta,4)$, where the variance is known to be 4 square units and $\theta$ is the unknown mean. Given $x=7$, compare the likelihoods at $\theta=5$ and $\theta=9$. (Answer: they are equal; each is $\frac{1}{\sqrt{8\pi}}e^{-1/2}\approx0.1210$ per unit.)

What You'll Learn