Product Laws Under Independent Identically Distributed Sampling

From One Observation to a Whole Sample

A statistical model often begins by describing the distribution of one observation. But a learning procedure is applied to a dataset, so its probability statements concern the joint behavior of all observations. The link between these two descriptions is especially simple when observations are independent and have the same distribution: the sample law is the product of the one-observation law.

That conclusion uses two distinct assumptions. Independence makes probabilities factor across observations; identical distribution says that the same factor appears at every position in the sample. If observations are independent but have different distributions, a product law still exists, but its factors need not match. Keeping these claims separate prevents a common modeling error.

In supervised learning, one observation is commonly a pair \(Z_i=(X_i,Y_i)\), consisting of features and a response. The observation space is then the space of such pairs, and \(P\) is the distribution of one pair. The iid assumption concerns the pairs as whole observations; it does not by itself assert that features and responses within a pair are independent.

For a measurable space \((\mathcal Z,\mathcal A)\), the product space \(\mathcal Z^n\) is equipped with its product sigma-algebra. The joint distribution of \((Z_1,\ldots,Z_n)\) is a probability measure on that space. A product measure \(P_1\otimes\cdots\otimes P_n\) is the measure whose value on a measurable rectangle \(A_1\times\cdots\times A_n\) is \(\prod_i P_i(A_i)\). These rectangles determine the product measure uniquely.

The Product-Law Result

Proof. Let \(Q\) denote the joint distribution of \((Z_1,\ldots,Z_n)\). For any measurable rectangle \(A_1\times\cdots\times A_n\), independence gives $$ Q(A_1\times\cdots\times A_n) =P(Z_1\in A_1,\ldots,Z_n\in A_n) =\prod_{i=1}^n P_i(A_i). $$ The product measure \(P_1\otimes\cdots\otimes P_n\) has exactly the same value on each such rectangle. Both are probability measures, and measurable rectangles form a generating pi-system for the product sigma-algebra. The uniqueness theorem for probability measures therefore gives \(Q=P_1\otimes\cdots\otimes P_n\). If every \(P_i=P\), this is \(P^{\otimes n}\). Conversely, evaluating the product measure on rectangles gives the factorization defining independence, and taking all but one rectangle coordinate to be \(\mathcal Z\) shows that each marginal is \(P_i\). \(\square\)

The theorem says that iid sampling determines the distribution of the full ordered sample, not merely the distribution of each observation separately. In the iid case, every coordinate has the same law, and the product contains the same probability factor once per observation.

Mass Functions and Densities Under a Product Law

The measure statement has familiar forms when observations are discrete or have densities. If \(Z_i\) has probability mass function \(p_i\) on a countable space, then the joint mass function is the product of the marginal mass functions. If \(Z_i\) has density \(f_i\) with respect to a common measure, then the joint density is the product of the marginal densities. Thus, for iid observations, these expressions become powers of one common mass function or density.

Derivation. The function \(g(z_1,\ldots,z_n)=\prod_i f_i(z_i)\) is nonnegative and measurable. By Tonelli's theorem, its integral over the whole product space is $$ \int_{\mathcal Z^n}g\,d\mu^{\otimes n} =\prod_{i=1}^n\int_{\mathcal Z}f_i\,d\mu =1. $$ For a measurable rectangle \(A_1\times\cdots\times A_n\), Tonelli's theorem also gives $$ \int_{A_1\times\cdots\times A_n}g\,d\mu^{\otimes n} =\prod_{i=1}^n\int_{A_i}f_i\,d\mu =\prod_{i=1}^nP_i(A_i). $$ This is the defining value of \(P_1\otimes\cdots\otimes P_n\) on rectangles. Since rectangles determine the measure, \(g\) is a density of the joint law. \(\square\)

Independence Does Not Mean Identical Distribution

If observations are independent but their marginal distributions differ, the joint law remains a product, but it is \(P_1\otimes\cdots\otimes P_n\), not generally \(P^{\otimes n}\) for one common \(P\). The positions in the sample can then have different probabilities or densities. Independence alone does not permit replacing those factors by a common one.

A related distinction arises in fixed-design settings. Responses may be independent across units while their distributions depend on the units' different fixed feature values. Such responses can have a product joint law without sharing one common marginal distribution. This is why “independent observations” and “iid observations” are not interchangeable descriptions.

Common Mistake and Practical Consequence

• Wrong reasoning: “The observations are independent, so the joint law is \(P^{\otimes n}\).” Independence gives a product of marginal laws, but it does not show those marginal laws are equal.
• Correction: First identify the marginal law at each sample position. Use \(P^{\otimes n}\) only when all positions share the same law \(P\); otherwise use \(P_1\otimes\cdots\otimes P_n\).
• Another error: Treating a product density as evidence of identical distribution. A density of the form \(\prod_i f_i(z_i)\) expresses independence; identical distribution additionally requires \(f_i=f\) for every \(i\), when densities are used.

The iid assumption is useful because it reduces a sample-level description to one observation law and makes all sample positions interchangeable in distribution. It is a substantive assumption about how the data were generated, not a consequence of collecting observations in one dataset. If the sampling design or population changes across observations, the product law may still apply under independence, but the common-factor simplification may fail.