Random-Design Sampling from a Joint Distribution

Random Design Begins With a Joint Law

In supervised learning, the features supplied to a prediction rule are often not fixed in advance. They are observed along with responses, and both are generated by a probability mechanism. Describing that mechanism starts with the joint distribution of a feature vector and its response. Its marginal distribution tells us where features occur; its conditional distribution tells us how responses behave at a given feature value.

This formulation is called random-design sampling because the observed feature values are themselves random. It differs from a setup in which feature values are specified and only responses are random. The distinction concerns how the data are generated, not whether a particular feature value happens to be known after the data are collected.

A random-design training sample of size \(n\) consists of pairs \((X_1,Y_1),\ldots,(X_n,Y_n)\) that are independent and have the same joint law \(P_{XY}\). Thus each observed feature vector is random, and its corresponding response is observed with it. Independence here is across the sampled pairs. It does not assert that \(X_i\) and \(Y_i\) are independent within a pair; in learning problems, their association is usually central.

The law \(P_{XY}\) is the basic population description. The feature marginal \(P_X\) describes how often different feature regions occur, while the conditional response law describes response variation at a feature value. Neither one alone generally determines the other: a marginal distribution discards information about the association between \(X\) and \(Y\).

Worked Examples: Reading a Joint Distribution

Marginalization and Conditional Laws

For any measurable sets \(A\subseteq\mathcal X\) and \(B\subseteq\mathcal Y\), the joint law assigns probability to the event that both \(X\in A\) and \(Y\in B\). Summing or integrating over all response values removes the response coordinate and gives the feature marginal:

This identity says that a joint probability can be obtained by averaging the conditional probability over the feature distribution. To see why the marginal identity holds, note that \(A\times\mathcal Y\) is exactly the event \(\{X\in A\}\), since \(Y\) must take some value in its response space. For the conditional identity, the defining property of a regular conditional distribution is that its integral over any feature event \(A\) gives the joint probability of that feature event and response event \(B\). Taking \(B=\mathcal Y\) recovers \(P_X(A)\), because \(P_{Y\mid X}(\mathcal Y\mid x)=1\) for \(P_X\)-almost every \(x\).

The phrase almost every \(x\) with respect to \(P_X\) matters. A conditional distribution is identified by the joint law only up to changes on a set of feature values having \(P_X\)-probability zero. At such values, the observed population distribution provides no constraint on what conditional law to assign. When \(\mathcal X\) and \(\mathcal Y\) are standard Borel spaces, a probability-kernel version exists; this is the usual setting for real-valued and finite-valued learning data.

Conditional Densities and Their Ratio

When a joint density exists, the conditional density is often obtained by dividing by the feature marginal density. This formula is useful, but its domain must be stated carefully. Suppose \(X\) and \(Y\) have a joint density \(f_{XY}(x,y)\) with respect to a product of reference measures \(\mu(dx)\nu(dy)\). Define the marginal density, allowing the integral initially to be extended-valued, by

Because the joint density integrates to one, Tonelli's theorem gives \(\int f_X(x)\,\mu(dx)=1\). Consequently \(f_X\) is finite for \(\mu\)-almost every \(x\). Moreover, the set where \(f_X(x)=0\) has \(P_X\)-probability zero, and the set where \(f_X(x)=\infty\) also has \(P_X\)-probability zero. The ratio formula is therefore a statement for \(P_X\)-almost every feature value, not a pointwise assertion at every possible \(x\).

Here is the normalization and joint-law calculation. At any \(x\) where \(0<f_X(x)<\infty\) and the marginal integral formula holds,

For measurable \(A\subseteq\mathcal X\) and \(B\subseteq\mathcal Y\), integrate the proposed conditional probabilities against the feature marginal:

The cancellation is used only where \(0<f_X(x)<\infty\), a set of full \(P_X\)-probability. Nonnegative integration justifies the iterated integrals. Values of a conditional density on the exceptional \(P_X\)-null feature set can be assigned through any suitable version of the conditional kernel; they do not change the displayed identity. This qualification prevents division by zero, infinity, or an undefined \(\infty/\infty\).

Why the Distinction Matters

The joint-law formulation keeps three questions separate: where features occur, how responses behave at a given feature value, and how these quantities combine to generate observed pairs. This is important when specifying a learning target or assessing what a sample can represent. For instance, a conditional response law at a feature value that has zero probability under \(P_X\) is not determined by the observed joint law alone.

A common mistake is to reason that because the sampled pairs are independent, the feature and response within each pair must also be independent. The first statement concerns different indices \(i\) and \(j\); the second concerns \(X_i\) and \(Y_i\). In the normal example, pairs can be sampled independently from one another even though the conditional mean of \(Y_i\) is \(2X_i\), so the two coordinates within a pair are dependent.

Another mistake is to treat \(P_{Y\mid X}(\,\cdot\mid x)\) as uniquely determined at every imaginable \(x\). It is determined only \(P_X\)-almost everywhere. In a density calculation, asserting the ratio at a point with \(f_X(x)=0\), or with \(f_X(x)=\infty\), is not justified by the joint density formula.