Product Likelihoods Under Independent Sampling

From Individual Contributions to a Sample Likelihood

A likelihood describes how compatible an observed outcome is with each possible parameter value. For one observation, it is the probability mass or density evaluated at the observed value, viewed as a function of the parameter. With several responses, the relevant quantity is the joint probability model evaluated at the entire observed vector. Independence is the condition that lets this joint contribution factor into one contribution per response.

The responses need not have the same distribution for this factorization to hold. What matters here is independence; the distributions may differ from one response to another. The following results use a common parameter symbol $\theta$, but the model for each response may also depend on known, response-specific features.

Definition: Let $X_1,\ldots,X_n$ be independent responses with specified probability mass functions $f_i(x_i;\theta)$, or specified densities $f_i(x_i;\theta)$ with respect to their respective reference measures. For observed values $x_1,\ldots,x_n$, the product likelihood is $$ L(\theta;x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i;\theta). $$ For discrete responses, this is the joint probability mass at the observed vector. For continuous responses, it is a version of the joint density evaluated at that vector. In either case, the observed values are held fixed and the expression is considered as a function of $\theta$.

The same notation $f_i$ can denote a mass function or a density; the model makes clear which one is meant. In the continuous case, the density is specified as a particular function, since changing its values on a set of measure zero can change its value at a particular observed point. The product of the specified marginal densities is a version of the joint density under independence.

Why Independence Gives a Product

For discrete responses, independence directly says that the probability of a joint outcome is the product of its marginal probabilities. For continuous responses, the corresponding statement is about densities: the product of marginal densities is a joint density. The distinction matters because a density value is not itself the probability of observing an exact point.

Theorem: Suppose $X_1,\ldots,X_n$ are independent. If each response is discrete with mass function $f_i$, then their joint mass function is $$ f(x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i). $$ If response $X_i$ has density $f_i$ with respect to a measure $\mu_i$, then their joint distribution has density $\prod_{i=1}^n f_i(x_i)$ with respect to the product measure $\mu_1\otimes\cdots\otimes\mu_n$. Consequently, for a specified model indexed by $\theta$, the product likelihood is the joint mass or a version of the joint density evaluated at the observed vector.

Proof for discrete responses. By independence, for any values $x_1,\ldots,x_n$,

P(X_1=x_1,\ldots,X_n=x_n) =\prod_{i=1}^n P(X_i=x_i) =\prod_{i=1}^n f_i(x_i).

This is exactly the joint mass function at the specified vector.

Proof for densities. Assume the response spaces are equipped with the sigma-algebras on which the stated probability measures and product measure are defined. Let $P_i$ be the distribution of $X_i$, with density $f_i$ with respect to $\mu_i$. Independence means the joint distribution is the product measure $P_1\otimes\cdots\otimes P_n$. The function $F(x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i)$ is measurable and nonnegative. For any measurable rectangle $A_1\times\cdots\times A_n$, Tonelli's theorem gives

\begin{aligned} \int_{A_1\times\cdots\times A_n} F\,d(\mu_1\otimes\cdots\otimes\mu_n) &=\prod_{i=1}^n\int_{A_i}f_i\,d\mu_i\\ &=\prod_{i=1}^n P_i(A_i)\\ &=(P_1\otimes\cdots\otimes P_n)(A_1\times\cdots\times A_n). \end{aligned}

Both sides define finite measures, and measurable rectangles generate the product sigma-algebra. The measures therefore agree on the product sigma-algebra. Thus $F$ is a density of the joint distribution, proving the claim. Evaluating this product version at the observed vector gives the stated product likelihood. $\square$

Worked Discrete Examples

Worked Example: Independent Bernoulli Responses

Let $X_1,X_2,X_3$ be independent binary responses with success probabilities $\theta$, $\theta/2$, and $(1+\theta)/2$, respectively, where $0<\theta<1$. The responses are independent but have different probability models. Suppose the observed vector is $(1,0,1)$.

For a Bernoulli response with success probability $q$, the mass at $x\in\{0,1\}$ is $q^x(1-q)^{1-x}$. The first response contributes $\theta$, because its observed value is 1. The second contributes $1-\theta/2$, because its observed value is 0. The third contributes $(1+\theta)/2$, because its observed value is 1. Independence gives

L(\theta;1,0,1) =\theta\left(1-\frac{\theta}{2}\right)\frac{1+\theta}{2}.

For example, at $\theta=0.4$, the three contributions are $0.4$, $0.8$, and $0.7$, so

$$ L(0.4;1,0,1)=0.4(0.8)(0.7)=0.224. $$

This is the joint probability of the observed vector under that parameter value. Each factor comes from the model for its own response; independence, not identical distributions, justifies multiplying them.

Worked Example: Independent Binomial Counts

Let $X_1$ and $X_2$ be independent counts with $X_1\sim\operatorname{Binomial}(2,p)$ and $X_2\sim\operatorname{Binomial}(3,p)$, where $0<p<1$. Suppose the observed counts are $x_1=1$ and $x_2=2$. For a Binomial count with $m$ trials, the mass at $x$ is $\binom{m}{x}p^x(1-p)^{m-x}$. Thus the first contribution is

f_1(1;p)=\binom{2}{1}p(1-p)=2p(1-p),

and the second is

f_2(2;p)=\binom{3}{2}p^2(1-p)=3p^2(1-p).

The joint likelihood is their product:

L(p;1,2)=\bigl[2p(1-p)\bigr]\bigl[3p^2(1-p)\bigr] =6p^3(1-p)^2.

At $p=1/2$, this is $6(1/2)^3(1/2)^2=6/32=3/16$. The two counts have different numbers of trials, but the product follows from their stated independence.

Continuous Responses and the Log-Likelihood

Worked Example: Two Independent Normal Measurements

Suppose $X_1$ and $X_2$ are independent, with $X_1\sim N(\mu,1)$ and $X_2\sim N(\mu,4)$, where $\mu$ is unknown. Observe $x_1=0$ and $x_2=1$. The normal density with mean $\mu$ and variance $\sigma^2$ is

f(x;\mu)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}.

For $X_1$, the variance is 1, so its contribution at zero is

f_1(0;\mu)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\mu^2}{2}\right).

For $X_2$, the variance is 4, so its contribution at one is

f_2(1;\mu)=\frac{1}{\sqrt{8\pi}}\exp\left(-\frac{(1-\mu)^2}{8}\right).

Multiplication gives the joint density version and hence the likelihood:

L(\mu;0,1) =\frac{1}{4\pi} \exp\left\{-\frac{\mu^2}{2}-\frac{(1-\mu)^2}{8}\right\}.

At $\mu=0$, the exponent is $-0-(1)^2/8=-1/8$, and $L(0;0,1)=e^{-1/8}/(4\pi)$. This is a density value, not the probability of observing the exact pair $(0,1)$.

When each contribution is positive and finite, taking logarithms converts the product into a sum. This is useful because sums are often easier to manipulate and compute than products.

Result: If $0<f_i(x_i;\theta)<\infty$ for every $i$, then the log-likelihood satisfies $$ \ell(\theta;x_1,\ldots,x_n) =\log L(\theta;x_1,\ldots,x_n) =\sum_{i=1}^n\log f_i(x_i;\theta). $$ If all contributions are finite and at least one is zero, the product likelihood is zero and the extended log-likelihood is $-\infty$. These statements do not cover a product containing both zero and an infinite contribution.

Derivation. Under the stated positive-finite condition, the product likelihood is positive and finite. The logarithm is therefore defined, and the identity $\log(ab)=\log a+\log b$, applied repeatedly to the finite product, gives

\log\left(\prod_{i=1}^n f_i(x_i;\theta)\right) =\sum_{i=1}^n\log f_i(x_i;\theta).

If every contribution is finite and at least one is zero, multiplying the finite nonnegative contributions gives zero. With the convention $\log 0=-\infty$, the log-likelihood is $-\infty$; equivalently, the sum has at least one $-\infty$ term and all its other terms are finite or $-\infty$. If some contribution is infinite, however, this argument does not apply: a zero factor and an infinite factor produce the undefined expression $0\cdot\infty$, and the corresponding log sum would contain $-\infty+\infty$. $\square$

For the Normal example, all contributions are positive and finite. Its log-likelihood is consequently

\ell(\mu;0,1) =-\log(4\pi)-\frac{\mu^2}{2}-\frac{(1-\mu)^2}{8}.

The product-to-sum identity applies directly. It should not be used without checking the values of the factors when a model allows densities that are zero or infinite at observed points.

Common Pitfall: Multiplying Without Checking the Factors

A common mistake is to state without qualification that a zero contribution makes the likelihood zero and the log-likelihood $-\infty$. This is correct when the other contributions are finite, but not for arbitrary density versions: a valid density can be infinite at an individual point. If one observed contribution is zero and another is infinite, the product contains $0\cdot\infty$, which is undefined. Similarly, adding their logarithms would involve $-\infty+\infty$, also undefined.

The correction is to use the ordinary product and sum identities when all observed contributions are positive and finite. If the contributions are all finite but some are zero, the likelihood is zero and its extended logarithm is $-\infty$. If infinite values occur, do not force a conclusion from the product or log-sum formulas; check the specified model and density versions at the observed data.

A second, simpler error is to assume that every factor must be the same function because the responses are independent. Independence gives the product of the individual contributions. It does not require those contributions to be identical; the Bernoulli example used three different success probabilities.

Key takeaway: Form the likelihood by evaluating each response's specified mass function or density at its observed value, then multiply those contributions when the responses are independent. Take logarithms term by term only when the factors make that operation well-defined.

Check Your Understanding

Use the product rule and its conditions to answer the following questions.

Two independent Bernoulli responses have success probabilities $p$ and $q$. Write the likelihood for observed values $1$ and $0$.
What assumption justifies multiplying two response contributions to obtain their joint contribution?
For independent continuous responses with densities $f_1$ and $f_2$, what does $f_1(x_1)f_2(x_2)$ represent?
State the condition under which the log of a product likelihood is the sum of the individual log contributions.
Why is it unsafe to conclude that a product is zero if one factor is zero but another factor is infinite?