Independent Samples Versus Identically Distributed Samples

Two Different Assumptions

When responses are collected from several observational units, two questions must be kept separate: are the responses independent, and do they have the same distribution? Independence concerns how the responses relate to one another. Identical distribution concerns whether each response has the same probability model. One property does not imply the other.

The distinction matters in a generalized linear model because it determines the form of the joint response model and its likelihood. Independent responses can have different distributions, while responses that share one distribution need not be independent. A likelihood written as a product uses independence; a likelihood that repeats one common response model also uses identical distribution.

Definition: Random variables \(X_1,\ldots,X_n\) are independent if, for every choice of measurable sets \(A_1,\ldots,A_n\), \(P(X_1\in A_1,\ldots,X_n\in A_n)=\prod_{i=1}^n P(X_i\in A_i)\). They are identically distributed if each \(X_i\) has the same marginal probability distribution. They are independent and identically distributed (i.i.d.) if both conditions hold.

Identically distributed does not mean that the observed values are equal. For example, two independent responses can both follow a Bernoulli distribution with success probability \(0.4\), yet one response may be 0 and the other 1. Conversely, two independent responses can have different success probabilities and therefore different distributions.

What Independence Gives the Likelihood

Suppose \(X_i\) has density or mass function \(f_i\) with respect to a measure \(\mu_i\), and the \(X_i\) are independent. The measures and functions may differ across observations. The product likelihood follows from independence, whether or not the responses are identically distributed.

Theorem: If \(X_1,\ldots,X_n\) are independent and each has density \(f_i\) with respect to a measure \(\mu_i\), then their joint density with respect to the product measure \(\mu_1\otimes\cdots\otimes\mu_n\) is \(f(x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i)\). Thus, for observed values \(x_1,\ldots,x_n\), the likelihood is the product of the individual contributions, using each observation’s own model.

Proof. Define \(g(x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i)\). This is nonnegative. For measurable sets \(A_1,\ldots,A_n\), Tonelli’s theorem for nonnegative functions gives

\int_{A_1\times\cdots\times A_n} g\,d(\mu_1\otimes\cdots\otimes\mu_n) = \prod_{i=1}^n \int_{A_i} f_i\,d\mu_i = \prod_{i=1}^n P(X_i\in A_i).

By independence, the right-hand side is \(P(X_1\in A_1,\ldots,X_n\in A_n)\). Since measurable rectangles determine the joint distribution on the product space, \(g\) is a density of that joint distribution. Evaluating it at the observed values gives the stated product likelihood. \(\square\)

If the models depend on parameters, write \(f_i(x_i;\theta_i)\) when observation \(i\) has its own parameter, or \(f_i(x_i;\theta)\) when the observations share a parameter but may still have different model forms. Independence gives

L(\theta_1,\ldots,\theta_n)=\prod_{i=1}^n f_i(x_i;\theta_i), \qquad L(\theta)=\prod_{i=1}^n f_i(x_i;\theta),

in these two cases, respectively. The second product does not by itself imply identical distribution: the functions \(f_i\) can differ even when they contain a common parameter \(\theta\).

What Identical Distribution Adds

When the observations are independent and identically distributed, each response uses one common distribution. If its density or mass function is \(f(\,\cdot\,;\theta)\), then the joint density or mass function is the product of repeated copies of that same function.

Result: If \(X_1,\ldots,X_n\) are i.i.d. with density or mass function \(f(\,\cdot\,;\theta)\), their likelihood is \(L(\theta)=\prod_{i=1}^n f(x_i;\theta)\), and their log-likelihood is \(\ell(\theta)=\sum_{i=1}^n \log f(x_i;\theta)\), wherever the likelihood is positive.

Derivation. Identical distribution means that every marginal density or mass function is \(f(\,\cdot\,;\theta)\). Independence, by the theorem above, makes the joint density the product of the marginal densities. Substituting the observed values gives \(L(\theta)=\prod_i f(x_i;\theta)\). Taking the logarithm of this product, when all factors are positive, gives \(\ell(\theta)=\sum_i\log f(x_i;\theta)\). This is the common-model case of the independent product likelihood, not a consequence of independence alone. \(\square\)

Worked Example: Independent Bernoulli Responses With Different Probabilities

Let \(X_1\) and \(X_2\) be independent Bernoulli responses, with \(P(X_1=1)=0.2\) and \(P(X_2=1)=0.8\). The observed responses are \(x_1=1\) and \(x_2=1\). Are the responses independent, identically distributed, or both? What is their joint likelihood contribution?

They are independent by assumption. They are not identically distributed, because their success probabilities differ. For independent Bernoulli responses, the probability of observing both successes is the product of their success probabilities:

P(X_1=1,X_2=1)=0.2\times 0.8=0.16.

If one incorrectly treated them as i.i.d. Bernoulli responses with success probability \(0.5\), the corresponding probability would be \(0.5\times0.5=0.25\), not \(0.16\). The product calculation is valid here because the responses are independent; replacing the two actual models with one common model is not justified.

Worked Example: An i.i.d. Bernoulli Likelihood

Suppose four independent responses are all Bernoulli with the same unknown success probability \(p\). The observed sequence is \((1,0,1,1)\). Write its likelihood and evaluate it at \(p=0.6\).

The sequence contains three successes and one failure. Independence gives a product, and identical distribution means the same Bernoulli model contributes at every observation:

L(p)=p(1-p)p\,p=p^3(1-p).

At \(p=0.6\),

L(0.6)=0.6^3(1-0.6)=0.216\times0.4=0.0864.

This is the likelihood value for the observed sequence at \(p=0.6\). In the common Bernoulli model, the product can be grouped by the number of successes because each observation uses the same \(p\). That grouping depends on the identical-distribution assumption as well as independence.

Common and Observation-Specific Bernoulli Models

For independent Bernoulli responses with success probabilities \(p_1,\ldots,p_n\), the probability of observing \(x_i\in\{0,1\}\) at observation \(i\) is \(p_i^{x_i}(1-p_i)^{1-x_i}\). Multiplying the contributions gives a likelihood that retains the observation-specific probabilities.

Result: For independent Bernoulli responses with probabilities \(p_i\), the likelihood for observed values \(x_1,\ldots,x_n\) is \(L=\prod_{i=1}^n p_i^{x_i}(1-p_i)^{1-x_i}\). If all \(p_i=p\), this reduces to \(L(p)=p^s(1-p)^{n-s}\), where \(s=\sum_i x_i\).

Derivation. A Bernoulli response equals 1 with probability \(p_i\) and 0 with probability \(1-p_i\). For either possible observed value, its contribution is \(p_i^{x_i}(1-p_i)^{1-x_i}\): setting \(x_i=1\) gives \(p_i\), and setting \(x_i=0\) gives \(1-p_i\). Independence gives the product over observations. If every probability equals \(p\), the product becomes

\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum_i x_i}(1-p)^{\sum_i(1-x_i)} = p^s(1-p)^{n-s}.

The exponent \(s\) counts the successes, and \(n-s\) counts the failures. This simplification cannot generally be made when the \(p_i\) differ: then the identity of the observations with successes matters to the product. \(\square\)

Worked Example: Why the Success Count May Not Be Enough

Consider two independent Bernoulli responses with fixed probabilities \(p_1=0.2\) and \(p_2=0.8\). Compare the likelihood contributions for the two observed sequences \((1,0)\) and \((0,1)\). Both have one success.

For \((1,0)\), observation 1 is a success and observation 2 is a failure:

L(1,0)=0.2(1-0.8)=0.2\times0.2=0.04.

For \((0,1)\), observation 1 is a failure and observation 2 is a success:

L(0,1)=(1-0.2)0.8=0.8\times0.8=0.64.

The sequences have the same success count, but their likelihoods differ because the probabilities depend on which observation succeeds. In an i.i.d. Bernoulli model, both sequences would instead have likelihood \(p(1-p)\). Pooling by the total number of successes is therefore not generally valid for independent responses with different models.

Common Mistake and Why the Distinction Matters

A common mistake is to see a product likelihood and conclude that the observations are identically distributed. The product follows from independence; it may contain different factors \(f_i(x_i;\theta)\). A related mistake is to assume that a common parameter automatically makes the distributions identical. A shared parameter can appear in different observation-specific distributions, so check whether the full marginal distributions—not just one parameter symbol—are the same.

Before using a common response model, ask two separate questions: are the responses independent, and do they have the same marginal distribution? If the answer to the first is yes but the second is no, retain each observation’s own likelihood contribution. If both answers are yes, the repeated common model leads to the i.i.d. product form. These distinctions keep the assumed joint model aligned with the data-generating assumptions.

Check Your Understanding

For each question, distinguish assumptions about dependence from assumptions about marginal distributions.

Two independent Bernoulli responses have success probabilities \(0.3\) and \(0.7\). Are they identically distributed? Write the probability of observing two successes.
State the two conditions required for responses to be i.i.d.
Four i.i.d. Bernoulli responses have three observed successes. Explain why their likelihood can be written using the success count.
Two independent Bernoulli sequences both have one success, but the observations have different success probabilities. Why might the sequences have different likelihoods?
Does a product likelihood by itself establish that observations are identically distributed? Explain.