Conditional Laws as Regression Targets

Why the Regression Target Can Be a Distribution

A regression analysis need not target only a conditional mean. For a covariate vector $W$ and a real-valued outcome $Y$, the conditional law of $Y$ given $W=w$ describes the probabilities of all possible outcome events at that covariate value. It can therefore retain features such as skewness, multiple modes, and tail probability that a single mean does not represent.

The target is not automatically a density. It is first a probability measure depending measurably on the covariates. A conditional cumulative distribution function is one way to represent that measure when the outcome is real; a conditional density is available only under an additional absolute-continuity condition. These distinctions matter both for defining the estimand and for interpreting what data can identify.

Conditional Laws as Probability Kernels

Let $W$ take values in a measurable space $\mathcal W$, let $Y$ take values in a measurable space $\mathcal Y$, and let $P_{W,Y}$ be their joint distribution. Assume $\mathcal W$ and $\mathcal Y$ are standard Borel spaces, as are Euclidean spaces with their Borel sets. This assumption ensures the existence of a regular conditional distribution.

Definition: A conditional law of $Y$ given $W$ is a probability kernel $K$ from $\mathcal W$ to $\mathcal Y$. Thus, for each $w$, $K(w,\cdot)$ is a probability measure on $\mathcal Y$, and for each measurable set $C\subseteq\mathcal Y$, the map $w\mapsto K(w,C)$ is measurable. It is a version of the conditional law if, for every measurable $A\subseteq\mathcal W$ and $C\subseteq\mathcal Y$, $$ P(W\in A,\;Y\in C)=\int_A K(w,C)\,P_W(dw), $$ where $P_W$ is the marginal distribution of $W$.

The map $w\mapsto K(w,\cdot)$ is the distribution-valued regression target. The identity in the definition is the disintegration identity: it says that the joint law can be recovered by first drawing $W$ from its marginal distribution and then drawing $Y$ from the conditional measure $K(W,\cdot)$.

Theorem (disintegration identity): Under the assumptions above, for every nonnegative measurable function $h:\mathcal W\times\mathcal Y\to[0,\infty]$, $$ E[h(W,Y)]=\int_{\mathcal W}\left\{\int_{\mathcal Y}h(w,y)\,K(w,dy)\right\}P_W(dw). $$ If $h(W,Y)$ is integrable, the same identity holds for signed $h$.

Proof. Define a measure $Q$ on the product space by $$ Q(B)=\int_{\mathcal W}K(w,B_w)\,P_W(dw), \qquad B_w=\{y:(w,y)\in B\}. $$ For a measurable $B$, the kernel measurability theorem ensures that $w\mapsto K(w,B_w)$ is measurable. Countable additivity of $Q$ follows from countable additivity of each $K(w,\cdot)$ and monotone convergence. For any measurable rectangle $A\times C$, $$ Q(A\times C)=\int_A K(w,C)\,P_W(dw)=P_{W,Y}(A\times C). $$ Measurable rectangles form a pi-system that generates the product sigma-algebra. Since $Q$ and $P_{W,Y}$ are finite measures agreeing on that pi-system, the measure-uniqueness theorem gives $Q=P_{W,Y}$ on the entire product sigma-algebra. This step is important: agreement on rectangles alone does not assert agreement on every measurable set without the uniqueness argument.

For a nonnegative measurable $h$, the equality of measures gives the integral identity first for indicator functions of measurable sets, then for nonnegative measurable simple functions by linearity, and finally for $h$ by monotone convergence. For integrable signed $h$, apply the result to its positive and negative parts and subtract their finite integrals. $\square$

Measures, CDFs, and Densities Are Not Interchangeable

When $Y$ is real-valued, the conditional law can be represented by its conditional CDF $F(y\mid w)=K(w,(-\infty,y])$. For each fixed $w$, this is a nondecreasing, right-continuous function of $y$, with limits zero and one at the two ends of the real line. Conversely, such a CDF determines a probability measure on the real line. The CDF records probabilities of half-lines; the measure $K(w,\cdot)$ is the more general object that assigns probabilities to all measurable outcome events.

A conditional density is a further representation, not a requirement in the definition. If $K(w,\cdot)$ is absolutely continuous with respect to Lebesgue measure for $P_W$-almost every $w$, it has a density $f(y\mid w)$ satisfying $$ K(w,C)=\int_C f(y\mid w)\,dy $$ for measurable $C$, for almost every $w$. The density is defined only up to Lebesgue-null changes in $y$ for each such $w$. If the conditional law has an atom or a singular component, it may have no Lebesgue density at all.

Worked Example: A Conditional Law, Its CDF, and Its Density

Suppose $W$ takes values $0$ and $1$, with $P(W=0)=0.4$ and $P(W=1)=0.6$, and $Y\mid W=0\sim N(0,1)$, while $Y\mid W=1\sim N(2,4)$. The second argument of $N$ is the variance, so the standard deviations are $1$ and $2$, respectively.

The conditional probability measures are $K(0,\cdot)=N(0,1)$ and $K(1,\cdot)=N(2,4)$. Writing $\Phi$ for the standard normal CDF, their conditional CDFs are $$ F(y\mid 0)=\Phi(y), \qquad F(y\mid 1)=\Phi\left(\frac{y-2}{2}\right). $$ The corresponding densities are $$ f(y\mid 0)=\frac{1}{\sqrt{2\pi}}e^{-y^2/2}, \qquad f(y\mid 1)=\frac{1}{2\sqrt{2\pi}}e^{-(y-2)^2/8}. $$ For example, $P(Y\leq 0\mid W=0)=\Phi(0)=0.5$, whereas $P(Y\leq 0\mid W=1)=\Phi(-1)$. These probabilities come from the conditional measures, and the CDFs or densities are representations of those measures.

The unconditional mean is $E[Y]=0.4(0)+0.6(2)=1.2$, but it does not specify either conditional law. In particular, the two conditional distributions have different locations and spreads even though a single unconditional mean summarizes neither distinction.

Worked Example: An Atom Prevents a Lebesgue Density

For every $w\in[0,1]$, let $K(w,\cdot)$ put probability $0.3$ at zero and probability $0.7$ on an exponential distribution with rate $2$. For $y<0$, $F(y\mid w)=0$. For $y\geq0$, $$ F(y\mid w)=0.3+0.7(1-e^{-2y}). $$ In particular, $F(0\mid w)=0.3$: the law has an atom of size $0.3$ at zero.

If a Lebesgue density existed, the probability of the singleton $\{0\}$ would be the integral of that density over $\{0\}$. Since a singleton has Lebesgue measure zero, this integral would be zero. But $K(w,\{0\})=0.3$, a contradiction. The conditional measure and conditional CDF remain well-defined; only a Lebesgue density representation is unavailable.

Versions and What Is Identified

A conditional law is generally identified only $P_W$-almost everywhere. Values of a kernel at covariate values occurring with probability zero do not affect the disintegration identity. More precisely, any two regular conditional distributions for $Y$ given $W$ agree as probability measures for $P_W$-almost every $w$. Under the standard Borel assumption, this is equality outside one common $P_W$-null set, not merely equality separately for each outcome event.

Theorem (almost-everywhere uniqueness): If $K_1$ and $K_2$ are regular conditional distributions of $Y$ given $W$, with $\mathcal Y$ standard Borel, then there is a measurable set $N\subseteq\mathcal W$ with $P_W(N)=0$ such that $K_1(w,\cdot)=K_2(w,\cdot)$ for every $w\notin N$.

Proof. A standard Borel space has a countable pi-system $\mathcal C$ generating its sigma-algebra. For each $C\in\mathcal C$, both $K_1(W,C)$ and $K_2(W,C)$ are versions of the conditional probability of $Y\in C$ given $W$. Versions of the same conditional expectation agree almost surely, so there is a null set $N_C$ outside which $K_1(w,C)=K_2(w,C)$. Because $\mathcal C$ is countable, $N=\bigcup_{C\in\mathcal C}N_C$ is still null. For every $w\notin N$, the probability measures $K_1(w,\cdot)$ and $K_2(w,\cdot)$ agree on $\mathcal C$. The measure-uniqueness theorem then implies that they agree on the entire sigma-algebra of $\mathcal Y$. $\square$

Worked Example: A Point in the Support Can Still Be Null

Let $W$ be uniform on $[0,1]$, and suppose $K(w,\cdot)=N(w,1)$ on that interval. Define another kernel $K'$ to equal $K$ for every $w\neq 1/2$, but set $K'(1/2,\cdot)=\delta_5$, the point mass at $5$. Both kernels give the same integral against $P_W$, because $P(W=1/2)=0$. Thus both are versions of the same conditional law.

The point $1/2$ belongs to the topological support of the uniform distribution: every open interval around it has positive probability. Nevertheless, the joint distribution alone does not determine the conditional law at exactly $w=1/2$. A pointwise target there needs additional structure, such as a specified continuous version. Support membership is not the same as positive probability at the point.

Random-Design Targets and Support

In random-design prediction, new covariates are regarded as draws from a distribution, often $P_W$. A loss for predicting a distribution can therefore be averaged over both the covariates and outcomes. For a measurable predictor $Q(w)$, where $Q(w)$ is a probability measure on $\mathcal Y$, and a nonnegative measurable loss $\ell(Q(w),y)$, the risk has the form $$ \mathcal R(Q)=E[\ell(Q(W),Y)] =\int_{\mathcal W}\int_{\mathcal Y}\ell(Q(w),y)\,K(w,dy)\,P_W(dw). $$ The disintegration identity justifies the equality. The target $K$ in such a risk matters only up to $P_W$-almost-everywhere equality. Thus changing a predictor or a version on a covariate-null set does not change its random-design risk.

A different target weighting can answer a different question. If performance is averaged over a covariate distribution $\nu$ rather than $P_W$, the corresponding integral weights covariate regions according to $\nu$. When $\nu$ assigns positive mass to a set having zero $P_W$-probability, the observed joint law does not identify the conditional law there. Estimating such a target requires additional assumptions or data; it cannot be obtained from the random-design distribution alone.

Worked Example: Changing the Covariate Weighting Changes the Targeted Risk

Suppose $W\in\{0,1\}$, with $P(W=0)=0.9$ and $P(W=1)=0.1$. Consider an illustrative loss whose conditional expectation for a predictor $Q$ is $1$ at $w=0$ and $5$ at $w=1$. The random-design risk is $$ \mathcal R(Q)=0.9(1)+0.1(5)=1.4. $$

If the evaluation population instead assigns probability $0.5$ to each covariate value, the weighted risk is $0.5(1)+0.5(5)=3$. The conditional laws have not changed; the weighting over covariates has. This distinction is why an estimand should specify both the conditional distribution target and the covariate distribution over which performance is averaged.

Common Pitfall

A common error is to treat the conditional target as a uniquely specified density at every covariate value. That reasoning fails twice: a conditional law may contain atoms or singular components, and even when it has a density, its values at a covariate-null set are not determined by the joint distribution. A formula based on dividing a joint density by a covariate density is valid only when the relevant densities exist and the covariate density is positive where the formula is used; it does not settle the target at points where that denominator vanishes.

The reliable starting point is the probability kernel $K(w,\cdot)$, understood up to $P_W$-almost-everywhere equality. Then ask whether the outcome law admits a CDF or a density representation, and whether a pointwise or differently weighted target requires additional assumptions.

Key takeaway: The conditional law is the distribution-valued regression target. A CDF represents it for real outcomes, and a density represents it only under absolute continuity. The joint distribution identifies the target only almost everywhere under the covariate distribution, so support and evaluation weighting must be interpreted with care.

Check Your Understanding

Use the definitions and results above to answer the following questions.

What two properties make $K(w,\cdot)$ a probability kernel, and what identity connects it to the joint law?
Why can two conditional-law versions disagree at a covariate value in the support and still describe the same joint distribution?
Give a reason a conditional CDF may exist when a conditional Lebesgue density does not.
For what covariate weighting is a random-design risk unchanged by modifying a predictor on a $P_W$-null set?
What additional issue arises if an evaluation distribution assigns positive mass to covariate values having zero probability under the data-generating marginal $P_W$?