Why Independence Determines the Likelihood
When several observations are collected, a likelihood must describe the probability model for the whole observed sample, not just for one observation. Independence gives a direct way to build that model: the joint probability or density of the observed values is the product of their individual probabilities or densities. Whether the observations also have the same distribution determines whether those factors can be identical.
A sample is independent and identically distributed (iid) when its observations are independent and each has the same probability distribution. Independence alone does not imply identical distributions. For example, separate measurements may have different variances, or different trials may have different success probabilities. In those cases the likelihood is still a product, but its factors need not be copies of one another.
The observations \(x_1,\ldots,x_n\) are fixed once the data have been collected; \(\theta\) varies in the likelihood. In an iid sample, every factor is the same function \(g\), so the product becomes \(\prod_{i=1}^{n}g(x_i;\theta)\). If the observations are independent but not identically distributed, the factor for observation \(i\) is \(g_i(x_i;\theta)\), and the factors must retain their observation-specific forms.
For discrete observations, each factor is the probability of the observed value. For continuous observations, each factor is a density evaluated at the observed value. A density value is not the probability that a continuous random variable equals that exact value; probabilities are obtained by integrating a density over a set. The joint density product is nevertheless the correct likelihood for the observed continuous data.
Why the Product Formula Holds
The product form follows directly from the definition of independence. For discrete observations, independence says that the probability of any specified combination of values is the product of the marginal probabilities. For continuous observations, independence gives the corresponding factorization of the joint density. In either case, fixing the observed values leaves a product whose factors depend on the model parameter.
For each \(i\), identify the probability mass function or density \(g_i(x_i;\theta)\), including its support and parameter restrictions.
Factor the joint probability or density into the product of the individual probabilities or densities.
Substitute the observed values \(x_i\). The resulting expression, as a function of \(\theta\), is the likelihood.
For instance, in the discrete case, independence gives \(P(X_1=x_1,\ldots,X_n=x_n)=\prod_{i=1}^{n}P(X_i=x_i)\). If each probability is specified by the model as \(g_i(x_i;\theta)\), substituting these probabilities yields the product likelihood. For continuous observations, the joint density factorization gives \(f_{X_1,\ldots,X_n}(x_1,\ldots,x_n;\theta)=\prod_{i=1}^{n}f_i(x_i;\theta)\). This argument requires independence; without it, the joint distribution must be specified directly and need not factor into marginal terms.
Worked Examples
Worked Example: An Exact Sequence From Identical Bernoulli Trials
A hypothetical sensor is tested independently five times. Let \(X_i=1\) if it passes test \(i\), and \(X_i=0\) otherwise. Assume the tests are iid Bernoulli with pass probability \(\theta\), where \(0\leq\theta\leq1\). The recorded ordered sequence is \((1,0,1,1,0)\). Find its likelihood and evaluate it at \(\theta=0.4\).
Method and model. A Bernoulli observation has probability \(\theta\) of being 1 and probability \(1-\theta\) of being 0. Independence means the probability of the exact sequence is the product of the five corresponding probabilities. Identical distribution means the same \(\theta\) applies to every test.
The sequence contains three passes and two failures, so its likelihood is
At \(\theta=0.4\), the pass probability is \(0.4\) and the failure probability is \(1-0.4=0.6\). Therefore,
The value \(0.02304\) is the probability of this particular ordered sequence under the specified parameter value. As a check, the factors correspond to exactly three observed passes and two observed failures. This method avoids adding a count-based combinatorial factor when the data are the specific ordered sequence shown.
Worked Example: Independent Trials With Different Success Probabilities
A hypothetical quality system records three independent checks. Given a parameter \(0\leq\theta\leq1\), the probability of a positive result is \(\theta/2\) for check 1, \(\theta\) for check 2, and \((1+\theta)/2\) for check 3. The observed results are \((1,0,1)\). Write the likelihood and evaluate it at \(\theta=0.4\).
Method and model. Each result is Bernoulli, but the success probabilities differ by check. The observations are independent, not identically distributed. For an observed 1 use its success probability; for an observed 0 use one minus its success probability.
The three observed factors are, in order, \(\theta/2\), \(1-\theta\), and \((1+\theta)/2\). Hence
At \(\theta=0.4\), the success probabilities are \(0.4/2=0.2\), \(0.4\), and \((1+0.4)/2=0.7\). The observed second result is 0, so its probability is \(1-0.4=0.6\). Thus
The result agrees with the simplified expression: \((0.4)(1-0.4^2)/4=(0.4)(0.84)/4=0.084\). The common mistake this method avoids is treating the three checks as if they shared one success probability, which would assign the wrong factor to at least two observations.
Worked Example: Independent Continuous Measurements With Different Distributions
A hypothetical instrument produces two independent continuous readings. In the model, \(X_1\sim N(\theta,1)\) and \(X_2\sim N(2\theta,4)\), where \(\theta\) is real and the second argument of \(N\) denotes variance. The observed readings are \(x_1=2\) and \(x_2=4\). Find their joint density, expressed as a likelihood, at \(\theta=1\).
Method and model. The first reading has mean \(\theta\) and variance 1; the second has mean \(2\theta\) and variance 4. They are independent but not identically distributed. A normal density with mean \(\mu\) and variance \(\sigma^2\) is \(f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\{-\frac{(x-\mu)^2}{2\sigma^2}\}\).
At \(\theta=1\), the first reading has mean 1. Its density at \(x_1=2\) is
The second reading has mean \(2(1)=2\) and variance 4. Its density at \(x_2=4\) is
Independence gives the joint density and likelihood value:
The approximation is rounded to four decimal places. The joint density has units equal to the reciprocal of the product of the two measurement units. It is not the probability that both continuous readings equal exactly 2 and 4. The calculation avoids using one normal density twice, which would incorrectly impose identical distributions.
Identical Distribution Is an Additional Assumption
The product rule requires independence, not identical distributions. Identical distribution is a separate condition that lets us use one common probability mass function or density for every observation. The distinction is important whenever observations have different exposure levels, measurement precision, trial conditions, or model specifications.
| Sampling structure | Individual factors | Likelihood form |
|---|---|---|
| Independent and identically distributed | The same \(g\) applies to every observation | \(\prod_{i=1}^{n}g(x_i;\theta)\) |
| Independent, not identically distributed | Observation \(i\) uses its own \(g_i\) | \(\prod_{i=1}^{n}g_i(x_i;\theta)\) |
| Dependent observations | Marginal factors alone do not describe the joint model | Use the specified joint probability or density |
In an iid discrete model, repeated observed values can make the likelihood compact. For example, if a Bernoulli sample has \(k\) ones and \(n-k\) zeros, the likelihood of one particular ordered sequence with those outcomes is \(\theta^k(1-\theta)^{n-k}\). This simplification records how often each type of factor appears; it does not require the observations to be rearranged or treated as independent counts after the fact.
For independent nonidentical observations, even the same observed value can contribute different factors because its model differs by observation. Replacing \(\prod_i g_i(x_i;\theta)\) with a power of one common factor is valid only if the individual distributions truly coincide in the relevant model.
Common Mistake: Confusing an Ordered Sample With a Count
Suppose the observed data are the exact five-test sequence in the first example, \((1,0,1,1,0)\). A common incorrect calculation is \(10\theta^3(1-\theta)^2\), because there are \(\binom{5}{3}=10\) ways to place three passes among five positions. That expression is not the probability of the specific sequence observed; it sums the probabilities of all ten sequences with three passes.
If the recorded data are only the number of passes, namely 3 out of 5, then the count has a binomial distribution under the iid Bernoulli model, and its probability is \(\binom{5}{3}\theta^3(1-\theta)^2\). For an exact ordered sequence, the likelihood is just \(\theta^3(1-\theta)^2\). The distinction is the data being modeled: the sequence or the count.
Check Your Understanding
For each problem, identify the sampling structure and form the probability or density for the data as recorded.
- Four independent Bernoulli observations have common success probability \(p\). The recorded sequence is \((1,1,0,1)\). Write its likelihood and evaluate it at \(p=0.3\). (Answer: \(p^3(1-p)\); at \(p=0.3\), \(0.0189\).)
- Two independent Bernoulli observations have success probabilities \(\theta\) and \((1-\theta)/2\), respectively, where \(0\leq\theta\leq1\). The observed values are \((0,1)\). Write the likelihood and evaluate it at \(\theta=0.4\). (Answer: \((1-\theta)(1-\theta)/2\); at \(\theta=0.4\), \(0.18\).)
- Three independent checks have success probabilities \(0.2\), \(0.5\), and \(0.8\), with no unknown parameter. The exact observed sequence is \((1,0,1)\). Find its probability, and state whether the checks are identically distributed. (Answer: \(0.2(0.5)(0.8)=0.08\); they are independent but not identically distributed.)
- Independent measurements satisfy \(X_1\sim N(\theta,1)\) and \(X_2\sim N(\theta,9)\). For observed values \((1,4)\), write the likelihood at \(\theta=1\), leaving the exact expression unevaluated. (Answer: \(\frac{1}{\sqrt{2\pi}}\cdot\frac{1}{3\sqrt{2\pi}}\exp(-9/18)=\frac{e^{-1/2}}{6\pi}\).)
- Five iid Bernoulli trials produce exactly two successes, but their order is not recorded. Write the probability of the observed count when the success probability is \(p\), and explain why it differs from the likelihood of one specified ordered sequence with two successes. (Answer: \(\binom{5}{2}p^2(1-p)^3=10p^2(1-p)^3\); the count probability includes all 10 possible placements, while one ordered sequence has likelihood \(p^2(1-p)^3\).)