Random Samples as Independent Identically Distributed Observations

Why the Random-Sample Assumption Matters

Many statistical methods begin with repeated observations from one population. The mathematical model for those observations determines how their probabilities fit together. In particular, a common distribution for every observation is not enough: the observations must also be independent for the usual product formulas to apply.

A random sample is a sequence of independent observations that all have the same probability distribution. The two conditions—independence and identical distribution—work together. Identical distributions give each observation the same marginal law; independence specifies how probabilities for several observations combine. This tutorial develops those ideas through the joint law of the entire sample.

Definition: The random variables $X_1,\ldots,X_n$ form a random sample of size $n$ from a population distribution $F$ if they are independent and each has distribution $F$. Equivalently, for any measurable sets $A_1,\ldots,A_n$, $$ P(X_1\in A_1,\ldots,X_n\in A_n) =\prod_{j=1}^{n}P(X_j\in A_j), $$ and $P(X_j\in A)=F(A)$ for every $j$ and every measurable set $A$.

The notation $X_1,\ldots,X_n$ describes the random observations before they are observed. A particular data set consists of realized values, often written $x_1,\ldots,x_n$. A random sample does not mean that the realized values must be distinct or that they must look alike. It means that, under the model, each observation has the same distribution and the observations are independent.

How the Conditions Determine the Joint Law

The joint law describes probabilities involving two or more sample observations together. For a random sample, the probability of a rectangular event—one condition imposed separately on each observation—factors into the product of its marginal probabilities. For example, $P(X_1\in A_1,\ X_2\in A_2)$ is $P(X_1\in A_1)P(X_2\in A_2)$. The events need not be the same; independence applies to different conditions on different observations.

Let $F(t)=P(X_j\le t)$ be the common cumulative distribution function. Applying independence to the events $\{X_j\le t_j\}$ gives the joint cumulative distribution function.

F_{X_1,\ldots,X_n}(t_1,\ldots,t_n) =P(X_1\le t_1,\ldots,X_n\le t_n) =\prod_{j=1}^{n}F(t_j).

This identity is the joint distribution function form of the iid assumption. If the observations are discrete with common probability mass function $p(x)=P(X_j=x)$, their joint probability mass function is the product of the individual masses. If they are continuous with common density $f$, their joint density is the product of the individual densities.

p_{X_1,\ldots,X_n}(x_1,\ldots,x_n) =\prod_{j=1}^{n}p(x_j), \qquad f_{X_1,\ldots,X_n}(x_1,\ldots,x_n) =\prod_{j=1}^{n}f(x_j).

For the density formula, the joint density is zero wherever any $x_j$ lies outside the support of the population density. Within the product support, the product formula applies. These formulas are not justified merely because the observations have the same marginal distribution; independence is essential.

Worked Example: An Ordered Pattern of Independent Flags

A hypothetical quality-control system performs three independent checks. Each check flags an item with probability $0.30$, and otherwise does not flag it. Let $X_j=1$ if check $j$ flags the item and $X_j=0$ otherwise. Find the probability of the ordered outcome “no flag, flag, flag.”

Model and method. The variables $X_1,X_2,X_3$ are iid Bernoulli with parameter $p=0.30$. Thus $P(X_j=1)=0.30$ and $P(X_j=0)=1-0.30=0.70$. Independence lets us multiply the probabilities in the specified order.

The requested event is $\{X_1=0,X_2=1,X_3=1\}$, so

$$ P(X_1=0,X_2=1,X_3=1) =(0.70)(0.30)(0.30) =0.0630. $$

The answer is $0.0630$, or $6.30\%$. As a check on the Bernoulli model, the probabilities of all possible ordered triples sum to $(0.70+0.30)^3=1$. The method avoids a common mistake: multiplying the probability of two flags by an unrelated count of possible orders when the question asks for one specified order.

A Justification of the Product Formulas

The joint cumulative distribution function formula follows directly from the definition of independence. For fixed thresholds $t_1,\ldots,t_n$, define $A_j=(-\infty,t_j]$. Independence says the probability that every observation lies in its corresponding set is the product of the individual probabilities. Since each observation has the same distribution $F$, the individual probability for observation $j$ is $F(t_j)$. Therefore,

\begin{aligned} P(X_1\le t_1,\ldots,X_n\le t_n) &=\prod_{j=1}^{n}P(X_j\le t_j)\\ &=\prod_{j=1}^{n}F(t_j). \end{aligned}

For discrete observations, set each $A_j$ to the single value $\{x_j\}$. The same independence definition gives the product of the individual point probabilities. For continuous observations with a joint density, the joint cumulative distribution function is the integral of that density over $(-\infty,t_1]\times\cdots\times(-\infty,t_n]$. A product of individual densities integrates over this rectangle to the product of the individual integrals, giving the displayed joint distribution function. Thus the product density is consistent with, and expresses, the independent common-density model.

Worked Example: A Joint Density and a Rectangular Event

Suppose $X_1,X_2,X_3$ are iid Uniform on $(0,2)$. Find the joint density and the probability of $X_1<0.5,\ X_2>1.5,\ 0.5<X_3<1$.

Model and method. Each variable has density $f(x)=1/2$ for $0<x<2$, and zero elsewhere. The observations are independent, so their joint density is the product of three individual densities. On the cube $0<x_1,x_2,x_3<2$,

f_{X_1,X_2,X_3}(x_1,x_2,x_3) =\left(\frac12\right)^3=\frac18.

For the event, each coordinate condition has probability equal to its interval length divided by the total interval length $2$:

\begin{aligned} P(X_1<0.5)&=\frac{0.5}{2}=\frac14,\\ P(X_2>1.5)&=\frac{2-1.5}{2}=\frac14,\\ P(0.5<X_3<1)&=\frac{1-0.5}{2}=\frac14. \end{aligned}

Independence now gives

P(X_1<0.5,\ X_2>1.5,\ 0.5<X_3<1) =\frac14\cdot\frac14\cdot\frac14 =\frac1{64} =0.015625.

To four decimal places, the probability is $0.0156$. As a check, integrating the joint density over the event gives $(1/8)(0.5)(0.5)(0.5)=1/64$, the same result. This method avoids treating a joint event as a single interval or forgetting that the second condition concerns the upper part of the support.

Same Distribution Does Not Mean Independent

A frequent modeling error is to infer independence because observations have identical marginal distributions. Sampling without replacement from a finite population illustrates why that inference fails. The first draw changes the composition of the population available for the second draw, creating dependence.

Worked Example: Sampling Two Objects Without Replacement

A hypothetical container holds two green tokens and two yellow tokens. Two tokens are drawn in order without replacement. Let $X_j=1$ if draw $j$ is green and $X_j=0$ if it is yellow. Determine whether $X_1,X_2$ form a random sample from a common distribution.

Marginal distributions. Each draw has probability $2/4=1/2$ of being green. For the second draw, this follows from symmetry: each of the four tokens is equally likely to occupy the second position. Thus both $X_1$ and $X_2$ are Bernoulli with parameter $1/2$; their marginal distributions are identical.

Joint probabilities. For two green draws, the first draw is green with probability $2/4$, and then one green token remains among three:

P(X_1=1,X_2=1)=\frac24\cdot\frac13=\frac16.

For a green first draw and a yellow second draw, two yellow tokens remain among the three:

P(X_1=1,X_2=0)=\frac24\cdot\frac23=\frac13.

By symmetry, $P(X_1=0,X_2=1)=1/3$ and $P(X_1=0,X_2=0)=1/6$. These four probabilities sum to $1/6+1/3+1/3+1/6=1$, as required.

If the draws were independent, the product of the marginal probabilities would give $P(X_1=1,X_2=1)=(1/2)(1/2)=1/4$, not $1/6$. Therefore, the observations have the same distribution but are dependent; they are not a random sample. With replacement, in contrast, the token composition resets before the second draw, and the two indicators would be independent Bernoulli variables with parameter $1/2$. The trap is to use the product of marginal probabilities without first verifying independence.

How to Check an iid Model

In an application, the iid assumption is a modeling statement about the data-generating process, not a conclusion that follows automatically from having a list of observations. Use the following checks before applying a joint product formula.

Identify the observation.
Specify what each $X_j$ measures and ensure that the observations are defined on comparable terms.

Check the common-distribution condition.
Ask whether each observation is modeled by the same population law, including the same parameters and measurement process.

Check independence.
Consider whether observing one value changes the probabilities for another. Dependence can arise from sampling without replacement, shared environmental conditions, or a common latent factor.

Apply the joint law only if justified.
When both conditions hold, multiply the appropriate marginal probabilities, masses, or densities. If either condition fails, the joint law needs a different model.

The assumptions are often reasonable for repeated measurements or selections from a large population when one observation has negligible effect on another. They may be poor approximations for clustered observations, repeated measurements on the same individual, or draws from a small finite population without replacement. Recognizing the distinction is important because later sampling-distribution calculations rely on the joint behavior of the observations, not just on their individual distributions.

Common Mistakes and Exam Traps

Using equal marginals as proof of independence. The token example has identical Bernoulli marginals but a joint probability different from the product.
Multiplying probabilities without an independence assumption. The product rule for a joint event applies when the relevant events are independent. Otherwise use conditional probabilities or the specified joint model.
Confusing an ordered outcome with an unordered description. The probability of one specified sequence is not automatically the probability of any sequence containing the same number of each outcome. If order is unrestricted, count the disjoint sequences or use an appropriate distribution.
Thinking iid observations must have different values. Independent draws can coincide; identical distribution describes their probability laws, not their realized values.
Ignoring the population mechanism. Drawing with replacement can produce independent observations, while drawing without replacement generally creates dependence, even when the marginal distributions match.

Key takeaway: A random sample requires both a common distribution and independence. Together, these conditions make the joint law a product of the marginal laws; neither condition alone is sufficient.

Check Your Understanding

For each problem, identify whether the product rule is justified before calculating.

Three independent devices each fail a test with probability $0.20$. Find the probability of the ordered pattern fail, pass, fail. (Answer: $0.0320$)
$X_1$ and $X_2$ are independent draws from a discrete population with $P(X=0)=0.20$, $P(X=1)=0.50$, and $P(X=2)=0.30$. Find $P(X_1=0,X_2=2)$. (Answer: $0.0600$)
$X_1$ and $X_2$ are iid Uniform on $(0,4)$. Find $P(X_1<1,\ 2<X_2<3)$. (Answer: $0.0625$)
A container holds three green and two yellow tokens. Two are drawn without replacement. Let $X_j=1$ for a green token and $0$ for a yellow token. Find $P(X_1=1,X_2=1)$, and state whether the draws are independent. (Answer: $3/10$; no, since the product of the marginal green probabilities is $9/25$)
$X_1,X_2$ are independent exponential random variables with rate $1.5$ per unit of time. Find $P(X_1>1,\ X_2\le1)$. (Answer: $e^{-1.5}(1-e^{-1.5})\approx0.1733$)

What You'll Learn