Binary Regression Functions and Label Distributions

Introduction

In binary prediction problems, each observation consists of features $X$ and a label $Y\in\{0,1\}$. The label might indicate whether a patient experiences an outcome, whether a message is spam, or whether a device fails. A useful way to describe how labels relate to features is to ask: at a given feature value, how likely is the positive label?

That conditional probability is the binary regression function. It describes the label distribution given the features, but it does not by itself describe how frequently different feature values occur. To specify the full distribution of the data, we need both the feature marginal distribution and the regression function.

The Binary Regression Function

Let $X$ take values in a feature space $\mathcal X$, and let $Y$ be binary, with positive label $1$ and negative label $0$. Assume $\mathcal X$ is a standard Borel space, a condition satisfied by common feature spaces such as finite sets and Euclidean spaces with their usual measurable sets. Write $P_X$ for the marginal distribution of $X$.

Definition: The binary regression function is a measurable function $\eta:\mathcal X\to[0,1]$ that represents the conditional probability of a positive label: $$ \eta(x)=P(Y=1\mid X=x). $$ More precisely, $\eta(X)$ is a version of the conditional expectation $E[\mathbf{1}_{\{Y=1\}}\mid \sigma(X)]$. The function is determined only up to sets of $P_X$-probability zero.

The precise formulation matters when $X$ is continuous. A particular feature value can have probability zero, so the elementary ratio $P(Y=1,X=x)/P(X=x)$ may be undefined. Conditional probability given $X=x$ is understood through a conditional distribution, which exists for the stated feature spaces. Equivalently, the definition says that for every measurable set $A\subseteq\mathcal X$,

P(X\in A,Y=1)=\int_A \eta(x)\,dP_X(x).

Since the two possible labels exhaust the label space, the conditional probability of label $0$ is $1-\eta(x)$, up to $P_X$-null sets. Thus the conditional distribution of $Y$ given $X=x$ is Bernoulli with success probability $\eta(x)$. For a binary variable, the conditional mean also equals this probability: $E[Y\mid X]=\eta(X)$.

Worked Example: A Feature With Three Possible Values

Suppose $X$ takes values $a,b,c$ with probabilities $0.2,0.5,0.3$, respectively, and $\eta(a)=0.1,\ \eta(b)=0.6,\ \eta(c)=0.9$. Calculate the joint probabilities of the feature and label, and the marginal probability of a positive label.

For each feature value, multiply its marginal probability by the conditional probability of each label. At $a$, $P(X=a,Y=1)=0.2(0.1)=0.02$ and $P(X=a,Y=0)=0.2(0.9)=0.18$. At $b$, the corresponding probabilities are $0.5(0.6)=0.30$ and $0.5(0.4)=0.20$. At $c$, they are $0.3(0.9)=0.27$ and $0.3(0.1)=0.03$.

Feature value	$P(X=x,Y=1)$	$P(X=x,Y=0)$	Row sum $P_X(x)$
$a$	0.02	0.18	0.20
$b$	0.30	0.20	0.50
$c$	0.27	0.03	0.30

Adding the positive-label entries gives $P(Y=1)=0.02+0.30+0.27=0.59$. The negative-label entries sum to $0.18+0.20+0.03=0.41$, and the full table sums to $1$, as required.

How the Two Pieces Specify the Joint Law

The marginal law $P_X$ determines where feature values occur. The function $\eta$ determines how the probability at each feature value is divided between labels $1$ and $0$. The following result gives the relationship exactly.

Theorem: Let $P$ be a joint distribution of $X\in\mathcal X$ and $Y\in\{0,1\}$, where $\mathcal X$ is standard Borel. If $P_X$ is the feature marginal and $\eta$ is a version of $P(Y=1\mid X=x)$, then for every measurable $A\subseteq\mathcal X$, $$ P(X\in A,Y=1)=\int_A\eta(x)\,dP_X(x),\qquad P(X\in A,Y=0)=\int_A(1-\eta(x))\,dP_X(x). $$ Consequently, $P_X$ and $\eta$, with $\eta$ specified up to $P_X$-null sets, determine the joint distribution of $(X,Y)$.

Proof. By the definition of conditional expectation, $E[\mathbf{1}_{\{X\in A\}}Y]=E[\mathbf{1}_{\{X\in A\}}E[Y\mid\sigma(X)]]$. Because $Y$ is binary, $Y=\mathbf{1}_{\{Y=1\}}$, and $E[Y\mid\sigma(X)]=\eta(X)$. Therefore

P(X\in A,Y=1) =E[\mathbf{1}_{\{X\in A\}}Y] =E[\mathbf{1}_{\{X\in A\}}\eta(X)] =\int_A\eta(x)\,dP_X(x).

Also, $\mathbf{1}_{\{Y=0\}}=1-Y$, so $E[\mathbf{1}_{\{Y=0\}}\mid\sigma(X)]=1-\eta(X)$. Applying the same conditional-expectation identity gives the formula for label $0$. Finally, any measurable event $C\subseteq\mathcal X\times\{0,1\}$ can be split into its two label sections $C_0=\{x:(x,0)\in C\}$ and $C_1=\{x:(x,1)\in C\}$. Its probability is the sum of the probabilities for those sections, each given by the displayed formulas. Hence the formulas determine the joint law. $\square$

The theorem also explains why changing $\eta$ on a set $N$ with $P_X(N)=0$ does not change the joint distribution: both integrals over $N$ are zero. Values assigned to $\eta$ at feature points that never occur are therefore immaterial to the data law.

Worked Example: A Continuous Feature

Let $X$ be uniform on $[0,1]$ and let $\eta(x)=x^2$. Find the joint densities for the two labels and the marginal label probabilities.

The feature density is $f_X(x)=1$ on $[0,1]$. For $y=1$, the joint density with respect to Lebesgue measure on the feature and counting measure on the label is $f_X(x)\eta(x)=x^2$. For $y=0$, it is $f_X(x)(1-\eta(x))=1-x^2$. Both densities are zero outside $[0,1]$.

Integrating the positive-label density gives $P(Y=1)=\int_0^1x^2\,dx=[x^3/3]_0^1=1/3$. For the negative label, $P(Y=0)=\int_0^1(1-x^2)\,dx=1-1/3=2/3$. Their sum is $1$. For instance, on a feature interval $A=[0,1/2]$, the joint positive-label probability is $P(X\in A,Y=1)=\int_0^{1/2}x^2\,dx=1/24$.

Constructing a Joint Distribution

The same relationship works in reverse: a feature distribution and a valid regression function can be used to build a joint law. This shows that the two ingredients are not merely summaries of a joint distribution; together they specify one.

Construction: Let $P_X$ be any probability distribution on $\mathcal X$, and let $\eta:\mathcal X\to[0,1]$ be measurable. Draw $X\sim P_X$, draw $U\sim\operatorname{Uniform}(0,1)$ independently of $X$, and set $Y=\mathbf{1}_{\{U\leq\eta(X)\}}$. Then $Y$ is binary and its conditional probability of being $1$, given $X=x$, is $\eta(x)$, for $P_X$-almost every $x$.

To verify the construction, condition on $X$. Independence makes $U$ uniform even after conditioning on $X$, so $P(Y=1\mid X)=P(U\leq\eta(X)\mid X)=\eta(X)$. For a measurable $A\subseteq\mathcal X$, the resulting joint probability is $P(X\in A,Y=1)=\int_A\eta(x)\,dP_X(x)$; replacing $\eta$ by $1-\eta$ gives the probability for $Y=0$. These probabilities are nonnegative and sum to $P_X(A)$. Thus they define the required joint distribution and recover the chosen feature marginal and regression function.

Worked Example: A Nonuniform Feature Distribution

Suppose $X$ has density $f_X(x)=2x$ on $[0,1]$, and $\eta(x)=1/4+x/2$. Compute the probability of a positive label and the probability that $X\leq1/2$ and $Y=1$. First, $2x$ integrates to $1$ on $[0,1]$, and $\eta(x)$ ranges from $1/4$ to $3/4$, so both are valid.

The joint density for a positive label is $f_X(x)\eta(x)=2x(1/4+x/2)=x/2+x^2$. Therefore $P(Y=1)=\int_0^1(x/2+x^2)\,dx=1/4+1/3=7/12$. The complementary probability is $P(Y=0)=1-7/12=5/12$.

For the restricted event, integrate only over $[0,1/2]$: $P(X\leq1/2,Y=1)=\int_0^{1/2}(x/2+x^2)\,dx=[x^2/4+x^3/3]_0^{1/2}=1/16+1/24=5/48$. As a check on the feature marginal, $P(X\leq1/2)=\int_0^{1/2}2x\,dx=1/4$. Thus the positive-label probability within this feature region is $(5/48)/(1/4)=5/12$, consistent with averaging $\eta(x)$ over that region under the conditional feature distribution.

What the Regression Function Does—and Does Not—Tell Us

The function $\eta$ describes labels conditional on features, not the frequency of the features themselves. Two populations can have the same $\eta$ but different feature marginals and therefore different joint distributions. For example, suppose the feature space is $\{a,b\}$ and $\eta(a)=0.2,\eta(b)=0.8$. One population might put probabilities $0.9$ and $0.1$ on $a$ and $b$; another might put probabilities $0.1$ and $0.9$. The conditional label probabilities agree, but the feature frequencies and overall positive-label probabilities differ: they are $0.9(0.2)+0.1(0.8)=0.26$ and $0.1(0.2)+0.9(0.8)=0.74$.

Conversely, knowing only $P_X$ is not enough to determine how labels are distributed among feature values. Fixing the same $P_X$ while changing $\eta$ generally changes the joint law. A full specification therefore consists of both parts, with the regression function considered only where the feature distribution places probability.

Pitfall: treating $\eta$ as the whole data distribution. This omits how often feature values occur. The correction is to specify $P_X$ as well as $\eta$.
Pitfall: using a pointwise ratio when $P(X=x)=0$. For continuous features, that ratio can be undefined. Use the conditional-distribution interpretation and the integral identity over measurable feature sets.
Pitfall: expecting unique values at every feature point. The joint law identifies $\eta$ only up to $P_X$-null sets. Different versions on such sets represent the same distribution.

Key takeaway: For binary labels, the joint law is obtained by combining the feature marginal $P_X$ with the conditional positive-label probability $\eta$. The positive and negative label probabilities over any feature set $A$ are the integrals of $\eta$ and $1-\eta$ over $A$, respectively.

Check Your Understanding

Use the definitions and identities above to answer these questions.

Why does a binary label imply that $E[Y\mid X]=P(Y=1\mid X)$?
If $P_X$ assigns probability $0.4$ to a feature value $a$, and $\eta(a)=0.25$, what are $P(X=a,Y=1)$ and $P(X=a,Y=0)$?
For continuous $X$, why can the ratio $P(Y=1,X=x)/P(X=x)$ fail to define $\eta(x)$?
Can two different feature marginals have the same regression function but different joint distributions? Explain.
Why does changing $\eta$ on a set of $P_X$-probability zero leave the joint law unchanged?