Likelihoods for Independent Econometric Samples

From a Sampling Model to a Likelihood

Likelihood is a way to express how well different parameter values accord with observed data under a specified model. In econometrics, observations often have different regressors, exposures, or other known inputs. Independence lets us construct the joint likelihood observation by observation, while preserving those differences.

The central distinction is between a probability distribution for random data and a likelihood function for parameters after the data have been observed. The same mathematical expression can describe both, but the role of its argument changes.

Definition: Suppose $Y_1,\ldots,Y_n$ are independent observations with probability mass functions or densities $f_i(y_i;\theta)$, where $\theta$ is an unknown parameter vector and any observation-specific inputs are treated as known. For observed values $y_1,\ldots,y_n$, the likelihood is $$ L(\theta;y_1,\ldots,y_n)=\prod_{i=1}^{n} f_i(y_i;\theta). $$ The log-likelihood is $\ell(\theta;y_1,\ldots,y_n)=\log L(\theta;y_1,\ldots,y_n)$ whenever the likelihood is positive; if the likelihood is zero, its log is $-\infty$.

Before observing the sample, the $Y_i$'s are random, and the joint probability law describes how possible data sets arise at a fixed $\theta$. After observing $y_1,\ldots,y_n$, those values are held fixed; $L$ is then viewed as a function of $\theta$. It is not a probability distribution over parameter values. In particular, a likelihood need not sum or integrate to one as $\theta$ varies.

For continuous observations, $f_i(y_i;\theta)$ is a density, not the probability of observing exactly $y_i$, which is ordinarily zero. A density, and therefore a likelihood based on continuous data, can depend on the measurement units and can be greater than one. Likelihood comparisons are meaningful within the stated model and for the same observed data.

Why Independence Produces a Product

Independence means that the joint probability mass function or density of the observations is the product of their individual mass functions or densities. Evaluating that joint function at the observed sample gives the likelihood. Taking its logarithm changes the product into a sum, which is often easier to work with.

\begin{aligned} f(y_1,\ldots,y_n;\theta) &=\prod_{i=1}^{n} f_i(y_i;\theta),\\ \ell(\theta;y_1,\ldots,y_n) &=\log\left(\prod_{i=1}^{n} f_i(y_i;\theta)\right) =\sum_{i=1}^{n}\log f_i(y_i;\theta). \end{aligned}

This factorization does not require identical distributions. For example, observations may have different known exposure durations or regressors, so their individual distributions differ while they remain independent conditional on those inputs. The product must then retain each observation’s own model contribution.

Worked Example: Independent Bernoulli Outcomes

Suppose five independent hypothetical binary outcomes are $y=(1,0,1,1,0)$. Assume each outcome has a Bernoulli distribution with the same success probability $p$, where $0\leq p\leq1$. Thus $P(Y_i=1)=p$, $P(Y_i=0)=1-p$, and the independence and common-probability assumptions specify the sampling model.

For an observed binary value $y_i$, its probability mass is $p^{y_i}(1-p)^{1-y_i}$. The likelihood is therefore

L(p;y)=\prod_{i=1}^{5}p^{y_i}(1-p)^{1-y_i} =p^3(1-p)^2.

There are three observed successes and two failures. At the hypothetical value $p=0.6$,

L(0.6;y)=(0.6)^3(0.4)^2 =0.216\times0.16 =0.03456.

For $0<p<1$, the log-likelihood is

\ell(p;y)=3\log p+2\log(1-p), \qquad \ell(0.6;y)=3\log(0.6)+2\log(0.4)\approx-3.365.

This is the likelihood of the observed sequence under $p=0.6$, not the probability that $p=0.6$. Because the sequence is discrete, here the likelihood value also equals its sampling probability at that parameter value.

Bernoulli Log-Likelihoods at the Boundaries

The Bernoulli parameter space includes $p=0$ and $p=1$, so a log-likelihood expression must be interpreted carefully there. The product form remains well-defined if a factor with exponent zero is taken to equal one, including at a base of zero. The logarithm is defined from the likelihood itself: $\log 0=-\infty$.

Writing $S=\sum_{i=1}^{n}y_i$, the likelihood is $p^S(1-p)^{n-S}$. Its expanded log form $S\log p+(n-S)\log(1-p)$ is directly valid for $0<p<1$. At the boundaries, use $\log L$ directly. Equivalently, the expanded expression can be interpreted by continuity with the convention $0\log 0=0$; this convention applies to a zero coefficient multiplying a log, not to the logarithm of a zero likelihood.

For example, if all $n$ observations are successes, then $S=n$ and $L(1;y)=1$, so $\ell(1;y)=0$. If the sample contains both successes and failures, then $L(0;y)=L(1;y)=0$, and the log-likelihood at either boundary is $-\infty$. This distinction prevents the expression $0\log(0)$ from being treated as an undefined contribution when the overall likelihood is positive.

Gaussian Likelihoods

Consider independent observations $Y_i\sim N(\mu,\sigma^2)$, with a common mean $\mu$ and common variance $\sigma^2>0$. The normal sampling density for each observation includes both its exponential term and its normalizing constant. Retaining the constant gives the full likelihood; it is not a function of $\mu$ or $\sigma^2$, but it is part of the density.

L(\mu,\sigma^2;y) =\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i-\mu)^2}{2\sigma^2}\right) =(2\pi\sigma^2)^{-n/2} \exp\left(-\frac{\sum_{i=1}^{n}(y_i-\mu)^2}{2\sigma^2}\right).

Taking logarithms gives

\ell(\mu,\sigma^2;y) =-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2.

Worked Example: Gaussian Density for Three Observations

Suppose three independent hypothetical measurements are $8$, $10$, and $12$ units, and assume each follows $N(\mu,\sigma^2)$ with $\mu=10$ units and $\sigma=2$ units, so $\sigma^2=4$ square units. The independent common-normal sampling assumption yields

\begin{aligned} L(10,4;y) &=(2\pi\cdot4)^{-3/2} \exp\left(-\frac{(8-10)^2+(10-10)^2+(12-10)^2}{2\cdot4}\right)\\ &=(8\pi)^{-3/2}\exp(-1)\\ &\approx 0.002920\ \text{units}^{-3}. \end{aligned}

The squared residuals sum to $4+0+4=8$, giving the exponent $-8/8=-1$. The corresponding log-likelihood is

\ell(10,4;y) =-\frac{3}{2}\log(2\pi)-\frac{3}{2}\log(4)-1 \approx -5.836.

The likelihood has inverse-cubic units because it is a joint density for three continuous measurements. It is not the probability of observing exactly these three values.

In regression applications, a Gaussian model may instead assign observation $i$ a mean depending on known regressors, such as $x_i^\mathsf{T}\beta$. If the observations are independent and $Y_i\sim N(x_i^\mathsf{T}\beta,\sigma^2)$, the product formula still applies, with $(y_i-\mu)$ replaced by $(y_i-x_i^\mathsf{T}\beta)$. The individual means need not be equal for independence to give the product.

Poisson Likelihoods With Different Exposures

Suppose count observations are independent conditional on known positive exposures $e_i$, with $Y_i\sim\operatorname{Poisson}(e_i\lambda)$ and common rate $\lambda>0$. The exposure scales the expected count, so $E(Y_i\mid e_i)=e_i\lambda$. The model assumes the stated Poisson sampling distribution and conditional independence; it does not assume that the counts have equal means when exposures differ.

\begin{aligned} L(\lambda;y,e) &=\prod_{i=1}^{n} \frac{\exp(-e_i\lambda)(e_i\lambda)^{y_i}}{y_i!}\\ &=\exp\left(-\lambda\sum_{i=1}^{n}e_i\right) \lambda^{\sum_{i=1}^{n}y_i} \prod_{i=1}^{n}\frac{e_i^{y_i}}{y_i!}. \end{aligned}

The corresponding log-likelihood, for positive $\lambda$, is

\ell(\lambda;y,e) =-\lambda\sum_{i=1}^{n}e_i +\left(\sum_{i=1}^{n}y_i\right)\log\lambda +\sum_{i=1}^{n}y_i\log e_i -\sum_{i=1}^{n}\log(y_i!).

Worked Example: Poisson Counts With Unequal Exposures

For three hypothetical independent counts, let the observed values be $y=(2,0,3)$ and the known exposures be $e=(1,2,1)$. Assume $Y_i\sim\operatorname{Poisson}(e_i\lambda)$ and evaluate the likelihood at the hypothetical rate $\lambda=1.5$ counts per exposure unit. The total exposure is $1+2+1=4$, and the total count is $2+0+3=5$.

\begin{aligned} L(1.5;y,e) &=\prod_{i=1}^{3} \frac{\exp(-e_i\cdot1.5)(e_i\cdot1.5)^{y_i}}{y_i!}\\ &=\exp(-4\cdot1.5)(1.5)^5 \left(\frac{1^2\,2^0\,1^3}{2!\,0!\,3!}\right)\\ &=\exp(-6)\frac{(1.5)^5}{12}\\ &\approx 0.001569. \end{aligned}

Here $0!=1$, so the zero count contributes a valid Poisson probability factor. The log-likelihood at this rate is

\ell(1.5;y,e) =-6+5\log(1.5)-\log(12) \approx -6.458.

The exposure values matter: the second count is modeled with mean $2\lambda$, not $\lambda$. The displayed product and its log therefore preserve the observation-specific sampling assumptions.

What May Be Omitted From a Likelihood?

When studying how a likelihood varies with $\theta$, a factor that is positive and does not depend on $\theta$ does not change relative likelihood comparisons over $\theta$. For example, the factorial terms in the Poisson likelihood are data-only factors. One may omit them when only the parameter-dependent shape is needed, but the resulting expression is then proportional to the full likelihood, rather than the full likelihood itself.

The distinction matters when reporting an actual likelihood value or log-likelihood. It also matters if the omitted quantity is not common to the comparison being made. A reliable practice is to first write the complete product from the sampling model, then identify explicitly any parameter-independent terms being set aside.

Common Mistakes and a Construction Checklist

A likelihood calculation is only as sound as the sampling assumptions that produce it. Independence justifies multiplying the individual contributions; it does not follow merely from having several observations. Likewise, a common parameter does not imply identical distributions when observations have different regressors or exposures.

Calling likelihood a probability for the parameter. The likelihood evaluates the data-generating mass or density at fixed observed data as a function of the parameter. It does not assign probabilities to parameter values.
Using a density as a point probability. For continuous data, the Gaussian likelihood is built from density values. The probability of one exact sample point is not that density value.
Logging before checking the parameter boundary. The Bernoulli expanded log formula is directly valid in the interior. At $p=0$ or $p=1$, calculate $\log L$ from the product, or apply the stated continuity convention carefully.
Dropping terms without saying so. A parameter-free factor may be omitted for particular relative comparisons, but it belongs to the full likelihood and can matter when reporting its value.

Key takeaway: State the sampling model and independence assumptions first. Evaluate each observation’s mass or density at its observed value, multiply those contributions to form the likelihood, and take logarithms to obtain an additive log-likelihood. Then keep clear which quantities are fixed data and which are parameters.

Check Your Understanding

Use the sampling assumptions in each question to construct or interpret the likelihood.

For independent Bernoulli observations with three successes and two failures, write the likelihood for $p$ and state its value at $p=0.4$.
Why is a Gaussian likelihood based on exact continuous observations not the probability of observing those exact values?
Suppose two independent Poisson counts have exposures $1$ and $3$. What are their means under a common rate $\lambda$, and why should their likelihood contributions differ?
For an all-success Bernoulli sample, what are the likelihood and log-likelihood at $p=1$? Explain why the expanded interior log formula needs care there.
Give one example of a data-only factor in a likelihood and explain when it may be omitted from an expression.