Marginalizing Multivariate Laws

Why Marginalization Matters

A multivariate law describes several random variables together. Often a question concerns only some of them: for example, the distribution of a response and a treatment indicator, without regard to other recorded variables. The distribution of the selected variables is called their marginal distribution. Marginalization removes the coordinates that are not being studied while preserving the probabilities assigned to the coordinates that remain.

The operation depends on how the joint law is represented. For a countable joint support, probabilities are added over omitted coordinate values. For a joint density, probability is integrated over omitted coordinates. A joint CDF also determines a marginal CDF by allowing the bounds on omitted coordinates to increase without limit. These formulas are related, but they should not be confused: not every multivariate law has a joint PMF or density.

The Marginal Law of a Subvector

Definition: Let $X=(X_1,\ldots,X_d)$ be a random vector and let $S=\{i_1,\ldots,i_k\}$ be a nonempty set of coordinate indices. The marginal law of $X_S=(X_{i_1},\ldots,X_{i_k})$ is the probability distribution $$ P(X_S\in A)=P(X\in \pi_S^{-1}(A)) $$ for every Borel set $A\subseteq\mathbb{R}^k$, where $\pi_S$ is the projection that keeps the coordinates indexed by $S$ and discards the others.

This definition applies to any joint law, including discrete, continuous, mixed, and singular laws. Summation and integration formulas are convenient ways to calculate the marginal when the joint law has the corresponding representation. In particular, one must not assume that a joint density exists merely because the variables are real-valued.

Marginal PMFs: Sum Over Omitted Values

Suppose $X=(X_S,X_{S^c})$ has a joint PMF $p(x_S,x_{S^c})$ on a countable support. The marginal PMF of $X_S$ is

p_S(x_S)=\sum_{x_{S^c}}p(x_S,x_{S^c}),

where the sum includes every value of the omitted coordinates that is in the joint support. Terms outside that support have probability zero. The same rule works when the kept subvector has several coordinates; only the omitted coordinates are summed out.

Worked Example: Marginalizing a Three-Variable PMF

Let $X,Y,Z$ take values in $\{0,1\}$, with joint PMF $$ p(x,y,z)=\frac{1+x+2y+3z}{32},\qquad x,y,z\in\{0,1\}. $$ Find the joint marginal PMF of $(X,Z)$.

For fixed $x,z$, sum over the two possible values of $y$:

p_{X,Z}(x,z) =\sum_{y=0}^{1}p(x,y,z) =\frac{1+x+3z}{32}+\frac{1+x+2+3z}{32} =\frac{4+2x+6z}{32}.

Thus $p_{X,Z}(0,0)=4/32$, $p_{X,Z}(0,1)=10/32$, $p_{X,Z}(1,0)=6/32$, and $p_{X,Z}(1,1)=12/32$. These probabilities sum to $(4+10+6+12)/32=1$, as required. The result is a joint PMF for the two retained coordinates; it is not a PMF for $X$ alone.

Marginal Densities: Integrate Over Omitted Coordinates

Suppose $X=(X_S,X_{S^c})$ has a joint density $f(x_S,x_{S^c})$ with respect to Lebesgue measure. A marginal density of $X_S$ is obtained by integrating the joint density over the omitted coordinates:

f_S(x_S)=\int_{\mathbb{R}^{d-k}} f(x_S,x_{S^c})\,dx_{S^c}.

This equality specifies a density up to changes on sets of Lebesgue measure zero, as is standard for densities. The integral is over the full space of omitted coordinates; where the joint density is zero, the integrand contributes nothing. In practical calculations, it is often more efficient to write the integral over the nonzero part of the support.

Theorem: If $f$ is a nonnegative joint density on $\mathbb{R}^d$ with integral one, then integrating it over omitted coordinates gives a marginal density for the retained subvector, up to equality almost everywhere.

Proof. Define $f_S(x_S)=\int f(x_S,x_{S^c})\,dx_{S^c}$. Since $f\geq 0$, Tonelli’s theorem permits changing the order of integration, even before finiteness of the inner integral is known. For any Borel set $A\subseteq\mathbb{R}^k$,

\begin{aligned} \int_A f_S(x_S)\,dx_S &=\int_A\int_{\mathbb{R}^{d-k}}f(x_S,x_{S^c})\,dx_{S^c}\,dx_S\\ &=\int_{\pi_S^{-1}(A)}f(x)\,dx\\ &=P(X_S\in A). \end{aligned}

Taking $A=\mathbb{R}^k$ shows $\int f_S(x_S)\,dx_S=1$. Thus $f_S$ is a density of the marginal law. $\square$

Worked Example: A Two-Coordinate Marginal From a Three-Dimensional Density

Suppose $(X,Y,Z)$ has density $$ f(x,y,z)=\frac{4}{5}(1+xy),\qquad 0<x<1,\quad 0<y<1,\quad 0<z<1, $$ and density zero elsewhere. First check its normalization, then find the marginal density of $(X,Z)$.

The integral of the unnormalized factor over the unit cube is $$ \int_0^1\int_0^1\int_0^1(1+xy)\,dz\,dy\,dx =1+\left(\int_0^1x\,dx\right)\left(\int_0^1y\,dy\right) =1+\frac{1}{2}\frac{1}{2} =\frac{5}{4}. $$ Multiplication by $4/5$ therefore gives total integral one.

To retain $X$ and $Z$, integrate out $Y$. For $0<x<1$ and $0<z<1$, $$ f_{X,Z}(x,z) =\int_0^1\frac{4}{5}(1+xy)\,dy =\frac{4}{5}\left(1+\frac{x}{2}\right) =\frac{4}{5}+\frac{2x}{5}. $$ The marginal density is zero outside the unit square. As a check, $$ \int_0^1\int_0^1\left(\frac{4}{5}+\frac{2x}{5}\right)\,dz\,dx =\frac{4}{5}+\frac{1}{5}=1. $$

Marginal CDFs From Joint CDFs

For a random vector $X=(X_1,\ldots,X_d)$, its joint CDF is $F_X(x_1,\ldots,x_d)=P(X_1\leq x_1,\ldots,X_d\leq x_d)$. The marginal CDF of a retained subvector can be recovered without differentiating or summing the joint CDF. Instead, send every omitted coordinate bound to $+\infty$.

Theorem: For $S=\{i_1,\ldots,i_k\}$, the marginal CDF is $$ F_{X_S}(x_{i_1},\ldots,x_{i_k}) =\lim_{n\to\infty}F_X(u_1^{(n)},\ldots,u_d^{(n)}), $$ where $u_{i_j}^{(n)}=x_{i_j}$ for retained coordinates and $u_r^{(n)}=n$ for omitted coordinates.

Proof. For each positive integer $n$, let $A_n$ be the event that every retained coordinate satisfies its specified bound and every omitted coordinate is at most $n$. The events $A_n$ increase with $n$. Their union is the event that all retained coordinates satisfy their bounds: every omitted real-valued random variable is finite with probability one, so for any outcome satisfying the retained bounds there is an integer $n$ large enough to exceed all its omitted coordinates. Continuity of probability for increasing events gives $P(A_n)\to P(\bigcup_n A_n)$. The left side is the joint CDF evaluated at the bounds in the theorem, and the limit is the marginal CDF. $\square$

Worked Example: Obtaining a Marginal CDF by Sending a Bound to Infinity

Let $(X,Y)$ have density $f(x,y)=2x$ on $0<x<1,\ 0<y<1$, and zero elsewhere. For $0\leq x\leq1$ and $0\leq y\leq1$, its joint CDF is $$ F_{X,Y}(x,y)=\int_0^x\int_0^y 2u\,dv\,du=x^2y. $$

To obtain the marginal CDF of $X$, set the $Y$-bound to $n$. For any integer $n\geq1$, the bound includes the entire support of $Y$, so $F_{X,Y}(x,n)=x^2$ when $0\leq x\leq1$. The full marginal CDF is therefore $$ F_X(x)= \begin{cases} 0,&x<0,\\ x^2,&0\leq x\leq1,\\ 1,&x>1. \end{cases} $$ Differentiating on the interval where the CDF is smooth gives the marginal density $f_X(x)=2x$ for $0<x<1$, and zero outside. It integrates to $\int_0^1 2x\,dx=1$.

Marginalizing Laws Without a Joint Density

The projection definition of a marginal remains valid when a joint density does not exist. For example, let $U$ be uniform on $(0,1)$ and set $X=U$, $Y=U$. The joint law is concentrated on the diagonal $x=y$, which has area zero in the plane, so it cannot be represented by an ordinary two-dimensional density. Nevertheless, both one-dimensional marginals are uniform on $(0,1)$: for every Borel set $A$, $P(X\in A)=P(U\in A)$ and $P(Y\in A)=P(U\in A)$.

This example also shows why the formula “integrate the joint density over the omitted coordinate” requires a joint density as a hypothesis. For a singular law, marginalization means taking the distribution under projection. If a law is represented using several components, such as point, curve, and area components, their probabilities must be projected according to that representation; an area integral alone would miss any non-area component.

Common Mistakes and Checks

Summing or integrating over only some omitted values. A marginal includes every omitted coordinate value allowed by the joint support. Leaving out valid values loses probability.
Using the joint support as the marginal support. The marginal support is obtained by projection. A retained value is possible if at least one omitted-coordinate value can accompany it with positive probability or density contribution.
Assuming every joint law has a density. Singular laws can have marginals with densities even when the full joint law has no density. Use the probability definition unless the required joint PMF or density exists.
Treating an infinite CDF bound as an ordinary finite evaluation. The rigorous statement is a limit as the omitted bounds increase; continuity of probability justifies that limit.
Forgetting the normalization check. A marginal PMF must sum to one, and a marginal density must integrate to one. These checks often expose a missing term or an incorrect integration range.

Marginal distributions answer questions about selected coordinates while averaging over, or otherwise removing, the rest of the joint description. They are essential for describing a subset of variables, but they do not by themselves preserve all information about how the coordinates occur together.

Check Your Understanding

Try these questions using the definitions and calculations above.

A joint PMF is given for $(X,Y,Z)$. Write the summation that produces the marginal PMF of $(X,Z)$, and state which coordinate is summed out.
For a joint density of $(X,Y,Z,W)$, write the integral that gives the marginal density of $(X,W)$. What does the integration region include?
Explain why integrating a joint density is not a valid marginalization method for $(U,U)$ when $U$ is continuous.
State how to recover the marginal CDF of $X_1$ from the joint CDF of $(X_1,X_2,X_3)$, and identify the probability principle that justifies the limit.
After calculating a marginal density, which integral should equal one, and why is this a useful diagnostic?