Class Probability Functions in Classification

From a Classification Response to a Probability Vector

In a classification problem, the response $Y$ takes one of a finite set of labels, while $X$ denotes the observed features. Knowing the feature values does not always determine the label: two observations with the same features can belong to different classes. The conditional probabilities of those labels describe the class composition associated with the features.

These probabilities form a natural population target for classification. They describe the conditional response distribution, rather than a single predicted label. To keep the definition precise, suppose that $Y$ takes values in $\{1,\ldots,K\}$, where $K$ is a positive integer, and that $X$ takes values in a standard Borel space, such as a finite set or a Euclidean space. Let $P_X$ denote the distribution of $X$.

Definition: For each class $k\in\{1,\ldots,K\}$, a class-probability function is a measurable function $\eta_k$ satisfying $$ \int_A \eta_k(x)\,P_X(dx)=P(Y=k,\ X\in A) $$ for every measurable feature set $A$. Equivalently, $\eta_k(X)$ is a version of the conditional expectation $E[\mathbf{1}\{Y=k\}\mid X]$, and is written $P(Y=k\mid X=x)$. The vector $$ \eta(x)=\bigl(\eta_1(x),\ldots,\eta_K(x)\bigr) $$ is the class-probability vector (or conditional class distribution).

The notation $P(Y=k\mid X=x)$ is a convenient way to display a function of $x$. In the definition, it is the integral identity that gives the function its meaning. In particular, the conditional probability is not simply a probability of an event after substituting a number for $X$; it is a version of a conditional distribution, defined up to sets of feature values having probability zero.

Properties of the Class-Probability Vector

For any observed feature value where the functions are defined, the coordinates are intended to describe probabilities over all possible labels. Consequently, they are nonnegative and add to one, at least almost surely under the distribution of the features.

Theorem: For a finite-valued response $Y$, the class-probability functions satisfy $$ \eta_k(X)\geq 0\quad (k=1,\ldots,K), \qquad \sum_{k=1}^K\eta_k(X)=1 \quad\text{almost surely}. $$ Moreover, each $\eta_k$ is determined uniquely $P_X$-almost everywhere: if $\tilde{\eta}_k$ is another function satisfying the defining integral identity, then $\eta_k(X)=\tilde{\eta}_k(X)$ almost surely.

Proof. By definition, $\eta_k(X)$ is a version of $E[\mathbf{1}\{Y=k\}\mid X]$. Conditional expectation preserves nonnegativity, since $\mathbf{1}\{Y=k\}\geq 0$; hence $\eta_k(X)\geq 0$ almost surely. Also, the class indicators sum to one: $$ \sum_{k=1}^K\mathbf{1}\{Y=k\}=1. $$ Linearity of conditional expectation therefore gives $$ \sum_{k=1}^K\eta_k(X) = E\left[\sum_{k=1}^K\mathbf{1}\{Y=k\}\,\middle|\,X\right] = E[1\mid X] = 1 $$ almost surely.

For uniqueness, let $\eta_k$ and $\tilde{\eta}_k$ both satisfy the defining identity, and set $h=\eta_k-\tilde{\eta}_k$. Then $\int_A h\,dP_X=0$ for every measurable $A$. In particular, take $A=\{x:h(x)>0\}$. The integral of a nonnegative function that is strictly positive on this set can equal zero only if the set has $P_X$-probability zero. Taking $A=\{x:h(x)<0\}$ and applying the same argument to $-h$ shows that this set also has probability zero. Thus $h(X)=0$ almost surely, proving uniqueness. $\square$

The set of vectors with nonnegative coordinates that sum to one is called the probability simplex. The theorem says that $\eta(X)$ belongs to this simplex almost surely. Values at feature points outside the support of $P_X$ do not affect the defining identity, so the functions need not have a uniquely determined value there. This is why the almost-sure qualification matters.

Result: The class probabilities average to the overall class proportions: $$ E[\eta_k(X)]=P(Y=k),\qquad k=1,\ldots,K. $$ Indeed, taking $A$ to be the entire feature space in the defining identity gives $E[\eta_k(X)]=P(Y=k,\ X\text{ is in the feature space})=P(Y=k)$.

This identity is a useful consistency check. Conditional class probabilities describe how the class proportions vary with the features; averaging over the feature distribution recovers the population proportion in each class.

Worked Calculations

Worked Example: Class Probabilities From Joint Counts

A data-generating distribution assigns the following joint counts to two feature groups and three classes. Treating the counts as relative frequencies defines the corresponding empirical joint distribution.

Feature group	Class 1	Class 2	Class 3	Total
U	30	10	10	50
V	10	20	20	50

For group U, the total is 50, so the conditional probabilities are the class counts divided by 50: $$ \eta(U)=\left(\frac{30}{50},\frac{10}{50},\frac{10}{50}\right)=(0.6,0.2,0.2). $$ They are nonnegative and sum to $1$. For group V, $$ \eta(V)=\left(\frac{10}{50},\frac{20}{50},\frac{20}{50}\right)=(0.2,0.4,0.4), $$ which also sums to $1$.

The feature groups are equally common, so the average class-probability vector is $$ \frac12(0.6,0.2,0.2)+\frac12(0.2,0.4,0.4)=(0.4,0.3,0.3). $$ The overall class counts are 40, 30, and 30 out of 100, giving exactly the same class proportions. This verifies the averaging identity in this example.

Worked Example: Conditional Probabilities From Normal Densities

Suppose there are two classes, with $P(Y=1)=0.4$ and $P(Y=2)=0.6$. Given the class, the feature $X$ has distributions $X\mid Y=1\sim N(0,1)$ and $X\mid Y=2\sim N(2,1)$. Find the class-probability vector at $x=0$.

The two conditional densities at zero are $$ f_{X\mid Y=1}(0)=\frac{1}{\sqrt{2\pi}}, \qquad f_{X\mid Y=2}(0)=\frac{1}{\sqrt{2\pi}}\exp(-2). $$ The second expression follows because the normal density with mean 2 has exponent $-(0-2)^2/2=-2$. Bayes' rule gives $$ \eta_1(0)= \frac{0.4 f_{X\mid Y=1}(0)} {0.4 f_{X\mid Y=1}(0)+0.6 f_{X\mid Y=2}(0)} = \frac{0.4}{0.4+0.6\exp(-2)} \approx 0.831. $$ The common factor $1/\sqrt{2\pi}$ cancels. The other class probability is $$ \eta_2(0)= \frac{0.6\exp(-2)}{0.4+0.6\exp(-2)} \approx 0.169. $$ The results sum to one. Although class 2 is more prevalent overall, a feature value near the mean of class 1 makes class 1 more probable conditionally.

Worked Example: A Multiclass Probability Model

Suppose a three-class conditional distribution is specified by $$ \eta_1(x)=\frac{\exp(x)}{1+\exp(x)+\exp(-x)},\quad \eta_2(x)=\frac{1}{1+\exp(x)+\exp(-x)},\quad \eta_3(x)=\frac{\exp(-x)}{1+\exp(x)+\exp(-x)}. $$ Evaluate the class-probability vector at $x=\log 2$.

At this feature value, $\exp(x)=2$ and $\exp(-x)=1/2$. The denominator is $1+2+1/2=7/2$. Therefore $$ \eta(\log 2) = \left(\frac{2}{7/2},\frac{1}{7/2},\frac{1/2}{7/2}\right) = \left(\frac47,\frac27,\frac17\right). $$ The coordinates are all nonnegative and their sum is $4/7+2/7+1/7=1$. Thus the model assigns class 1 the largest conditional probability at this feature value, while retaining the probabilities for all three classes.

Interpretation and a Common Pitfall

A class-probability function is a feature-indexed description of the conditional distribution of the response. Its value at $x$ is a vector, not a label: for example, $(0.6,0.2,0.2)$ says that the conditional probabilities of the three classes are 0.6, 0.2, and 0.2. The vector preserves uncertainty that would be lost by reporting only one class.

Common mistake: “At $x$, class 1 has probability 0.6, so the response must be class 1 with probability 0.6 for every observation having those features.” This treats a conditional probability as a guarantee about a particular observation. The correction is that $\eta_1(x)=0.6$ describes the conditional distribution among responses at feature value $x$: it does not determine the response of any one observation. It also does not specify the value of the function meaningfully at feature points that have probability zero under $P_X$.

A second useful distinction is between a class-probability function and an estimate of one. The function $\eta$ is defined by the joint distribution of $(X,Y)$. Data can be used to construct an estimate of it, but the estimate is not the definition of the population target. Keeping the two separate makes it clear whether a statement concerns the data-generating distribution or a procedure applied to a sample.

Key takeaway: In a $K$-class problem, the conditional response target can be represented by $\eta(x)=(P(Y=1\mid X=x),\ldots,P(Y=K\mid X=x))$. Its coordinates form a probability vector almost surely, and averaging each coordinate over $X$ recovers the corresponding overall class proportion.

Check Your Understanding

Use the definitions and results above to answer the following questions.

For a four-class response, what are the two properties that the class-probability vector must satisfy almost surely?
In the joint-count example, what is the probability of class 2 conditional on feature group V?
Why can two functions that differ at a feature point of probability zero both represent the same class-probability function?
If $E[\eta_3(X)]=0.15$, what does this imply about the overall probability of class 3?
In the normal-density example, why can class 2 be more common overall while class 1 is more probable at $x=0$?

Class Probability Functions in Classification

What You'll Learn