Conditional Response Distributions as Data-Generating Objects

Why the Conditional Response Law Matters

In supervised learning, an observation consists of features \(X\) and a response \(Y\). Knowing only the distribution of \(X\) does not tell us how responses behave. Knowing only a summary such as the average response at each feature value does not tell us the full response variation either. The conditional response distribution records the complete distribution of \(Y\) given \(X\), including its spread, asymmetry, and discrete or continuous structure.

Together, the feature distribution and the conditional response distribution specify the joint law of \((X,Y)\). This gives a useful way to describe the data-generating process: first draw the features from their population distribution, then draw the response according to its conditional law at those features. The construction describes the population law of an observation; independent repeated sampling from that law is a separate sampling assumption.

Definition and Interpretation

For precision, suppose \(X\) takes values in a Borel subset \(\mathcal X\) of a Euclidean space and \(Y\) takes values in a Borel subset \(\mathcal Y\). Write \(P_X\) for the marginal distribution of \(X\). These usual feature and response spaces ensure that a regular conditional distribution exists.

Definition: A conditional response distribution is a probability kernel \(K\) from \(\mathcal X\) to \(\mathcal Y\). For each \(x\), \(K(x,\cdot)\) is a probability distribution on \(\mathcal Y\); for each measurable set \(B\subseteq\mathcal Y\), \(x\mapsto K(x,B)\) is measurable. It is a version of the conditional law of \(Y\) given \(X\) if, for every measurable \(A\subseteq\mathcal X\) and \(B\subseteq\mathcal Y\),

\(P(X\in A,\;Y\in B)=\int_A K(x,B)\,P_X(dx).\)

The value \(K(x,B)\) is commonly written \(P(Y\in B\mid X=x)\). This notation does not require \(P(X=x)>0\). If \(X\) is continuous, individual feature values usually have probability zero; the conditional law is instead defined as a measurable version that satisfies the integral identity above.

A conditional law specifies probabilities for every measurable response set, not just a single summary. When a conditional density \(f_{Y\mid X}(y\mid x)\) exists, the kernel is given by integrating that density over response sets. But a conditional law may also be discrete, or may combine discrete and continuous components, so a density need not exist.

Worked Example: A Binary Feature and Response

Let \(X\) and \(Y\) both be binary. Suppose \(P_X(\{1\})=0.4\), and the conditional response probabilities are \(K(1,\{1\})=0.8\) and \(K(0,\{1\})=0.2\). Since each conditional distribution is a probability distribution, \(K(1,\{0\})=0.2\) and \(K(0,\{0\})=0.8\).

The feature marginal gives \(P(X=0)=1-0.4=0.6\). Applying the defining identity to each pair of singleton sets gives the four joint probabilities:

\begin{aligned} P(X=1,Y=1)&=0.4(0.8)=0.32,\\ P(X=1,Y=0)&=0.4(0.2)=0.08,\\ P(X=0,Y=1)&=0.6(0.2)=0.12,\\ P(X=0,Y=0)&=0.6(0.8)=0.48. \end{aligned}

They are nonnegative and sum to \(0.32+0.08+0.12+0.48=1\), as required. For example, the response marginal is \(P(Y=1)=0.32+0.12=0.44\). The feature marginal and the two conditional response distributions have determined every entry of the joint distribution.

From Conditional Laws to Joint Probabilities

For any measurable feature set \(A\) and response set \(B\), the joint probability is obtained by averaging the conditional probability over the feature distribution. This is the key calculation behind the definition:

P(X\in A,\;Y\in B) =\int_A P(Y\in B\mid X=x)\,P_X(dx).

The integral is an average with respect to the distribution of feature values, not an average over a uniform range unless \(P_X\) is uniform. The formula also applies when \(A\) or \(B\) includes many possible values, not just singletons.

Theorem: Suppose \(K\) is a conditional response distribution for \(Y\) given \(X\), and \(P_X\) is the feature distribution. Then \(P_X\) and \(K\) determine the joint distribution of \((X,Y)\) uniquely.

Proof. The conditional-law identity determines \(P(X\in A,Y\in B)\) for every measurable rectangle \(A\times B\), since its value must be \(\int_A K(x,B)\,P_X(dx)\). Thus any two joint distributions having the same \(P_X\) and the same kernel \(K\) agree on all measurable rectangles. Rectangles form a collection closed under finite intersections and generate the product Borel sigma-algebra. The uniqueness theorem for probability measures therefore implies that the two joint distributions agree on every measurable subset of \(\mathcal X\times\mathcal Y\). Hence the joint law is uniquely determined. \(\square\)

This is a determination result: it says what the pair \((P_X,K)\) implies. Conversely, a joint distribution on these spaces admits a regular conditional distribution \(K\), and the displayed identity holds. Thus the description can be used in either direction.

Worked Example: A Continuous Feature with a Normal Conditional Law

Let \(X\sim\operatorname{Uniform}(0,1)\), and suppose \(Y\mid X=x\sim N(2x,1)\), where the second normal parameter is the variance. The feature density is \(1\) for \(0<x<1\), and the conditional response density is \(f_{Y\mid X}(y\mid x)=\frac{1}{\sqrt{2\pi}}\exp\!\left(-\frac{(y-2x)^2}{2}\right)\).

The joint density is the feature density multiplied by the conditional density. Therefore, for \(0<x<1\),

f_{X,Y}(x,y) =f_X(x)f_{Y\mid X}(y\mid x) =\frac{1}{\sqrt{2\pi}}\exp\!\left(-\frac{(y-2x)^2}{2}\right).

It is zero for \(x\notin(0,1)\). As a check on the conditional interpretation, at \(x=0.75\) the conditional response distribution is \(N(1.5,1)\). If \(\Phi\) denotes the standard normal cumulative distribution function, then \(P(Y>1\mid X=0.75)=1-\Phi((1-1.5)/1)=\Phi(0.5)\approx0.6915\).

A joint event can be calculated by integrating the conditional probability over the feature values in the event. For example,

P(X\leq 0.5,\;Y>1) =\int_0^{0.5} P(Y>1\mid X=x)\,dx =\int_0^{0.5}\Phi(2x-1)\,dx.

The integral is the required joint probability. It averages the conditional tail probability over \(X\) in the specified interval; it is not just the tail probability at one selected feature value.

Conditional Laws Are Defined Up to Feature-Null Sets

The joint-law identity identifies conditional probabilities only where the feature distribution places probability. Values of \(K(x,\cdot)\) can be changed on a set of feature values having \(P_X\)-probability zero without changing any joint probability. This is not a defect: such changes do not alter the data law.

Theorem: If \(K_1\) and \(K_2\) are two regular conditional distributions of \(Y\) given \(X\), then \(K_1(x,\cdot)=K_2(x,\cdot)\) for \(P_X\)-almost every \(x\). For response spaces with a countable generating class, the equality holds simultaneously as equality of probability measures outside one \(P_X\)-null set.

Proof. Fix a set \(B\) in a countable class of response sets that is closed under finite intersections and generates the response sigma-algebra. For every measurable \(A\subseteq\mathcal X\), both kernels satisfy

\int_A K_1(x,B)\,P_X(dx) =P(X\in A,Y\in B) =\int_A K_2(x,B)\,P_X(dx).

If two measurable bounded functions have equal integrals over every measurable \(A\), they are equal almost everywhere: otherwise one of the sets where the first exceeds the second by at least \(1/m\), for some positive integer \(m\), would have positive measure and give unequal integrals. Thus \(K_1(x,B)=K_2(x,B)\) almost everywhere for this \(B\). The generating class is countable, so the union of the exceptional null sets over its members is still a null set. Outside that union, the two probability measures agree on the generating class; uniqueness of probability measures then gives \(K_1(x,\cdot)=K_2(x,\cdot)\). \(\square\)

Worked Example: Changing a Conditional Law at a Probability-Zero Feature Value

Let \(X\sim\operatorname{Uniform}(0,1)\), and let \(Y\mid X=x\sim N(x,1)\) for \(0<x<1\). A regular conditional distribution must also be specified at \(x=0\), even though \(P(X=0)=0\). Consider two versions: \(K_1(0,\cdot)=N(0,1)\) and \(K_2(0,\cdot)=N(100,1)\), with both versions equal to \(N(x,1)\) for \(0<x<1\).

For any measurable response set \(B\) and feature set \(A\), the two integrals \(\int_A K_1(x,B)\,P_X(dx)\) and \(\int_A K_2(x,B)\,P_X(dx)\) are equal. Their integrands can differ only at \(x=0\), which contributes zero because the uniform distribution assigns that singleton probability zero. The two versions therefore give the same joint law, despite their different assigned distributions at \(x=0\).

This does not license arbitrary changes on a set of positive feature probability. For example, changing the conditional law throughout an interval of positive length would generally change the integral for some joint event, and hence change the joint law.

Why This Representation Is Useful

The pair \((P_X,K)\) separates two aspects of a population: where feature values occur, and how responses vary at those feature values. In random-design sampling, both aspects contribute to the joint data law. Under independent and identically distributed sampling, the law of a full sample is then the product of this one-observation joint law. Changing either the feature distribution or the conditional response distribution can change the population being described.

This separation also clarifies what a model for responses is claiming. A model that specifies only a conditional mean does not, by itself, specify the conditional response distribution: many distributions can share the same mean. The full conditional law is the data-generating object when the goal is to describe all conditional response probabilities, not just one chosen summary.

Common Mistake

Wrong reasoning: “Since \(P(X=x)=0\), the expression \(P(Y\in B\mid X=x)\) is meaningless for continuous features.” Correction: Interpret it as a value of a regular conditional distribution. The function \(x\mapsto K(x,B)\) is defined as a measurable version, and its values are constrained by the joint-law identity, up to feature-null sets.
Wrong reasoning: “A conditional mean and the feature distribution determine the joint law.” Correction: A mean alone does not specify the conditional distribution. For instance, distinct distributions can have the same mean but different probabilities of \(Y\) exceeding a threshold.
Wrong reasoning: “The conditional probability at one feature value is the joint probability of that response event.” Correction: A joint event involving a feature region requires integrating conditional probabilities over that region with respect to \(P_X\).

Key takeaway: A conditional response distribution assigns a probability law to the response at each feature value. Integrated against the feature distribution, it determines every joint probability, and its values are identifiable only up to sets of feature probability zero.

Check Your Understanding

Answer each question using the definitions and identities in this tutorial.

What two pieces of information determine the joint law of a feature-response pair?
Why can a conditional response distribution be defined at a feature value that has probability zero?
If \(K(x,B)\) is the conditional probability of a response set \(B\), write the formula for \(P(X\in A,Y\in B)\).
Why can two versions of a conditional response distribution differ at a single point when the feature distribution is continuous?
Does knowing \(P_X\) and the conditional mean \(E[Y\mid X=x]\) generally determine the joint law? Explain.