Measurable Feature and Response Spaces

Why Feature and Response Spaces Need Measurable Structure

A statistical experiment records feature values and responses, then makes probability statements about them. For example, we may ask for the probability that a feature lies in a particular region, or that a response belongs to a particular set of labels. To define such probabilities, it is not enough to name the set of possible values: we must also specify which subsets count as measurable events.

This distinction matters even when the possible values look familiar. Real-valued measurements have a standard measurable structure, while a finite set of class labels has another simple standard structure. More generally, the measurable structure determines which questions about a feature or response can be expressed as events in the underlying probability model.

The set \(S\) specifies the possible values; \(\mathcal S\) specifies the subsets of values to which probability statements may directly refer. A measurable space by itself does not assign probabilities. For that, we use a probability space \((\Omega,\mathcal F,P)\): \(\Omega\) is the set of outcomes of the experiment, \(\mathcal F\) is a sigma-algebra of events, and \(P\) is a probability measure on \(\mathcal F\).

The requirement says that every measurable question about the value of \(X\) corresponds to an event in \(\mathcal F\). Only then can \(P(X\in A)\) be defined as \(P(X^{-1}(A))\). Thus the measurable structure is part of the specification of the experiment, not an optional decoration added after probabilities have been calculated.

Standard Feature and Response Spaces

For real-valued features and responses, the standard choice is the Borel sigma-algebra \(\mathcal B(\mathbb R)\), generated by the open subsets of the real line. For a real vector in \(\mathbb R^p\), the standard choice is likewise the Borel sigma-algebra \(\mathcal B(\mathbb R^p)\) generated by its usual topology. These choices make familiar sets, such as intervals and open regions, measurable.

For a finite label set \(\mathcal Y=\{1,\ldots,K\}\), the standard choice is its power set \(2^{\mathcal Y}\), containing every subset of labels. Then events such as “the response is class 1 or class 3” are measurable. The labels need not be numbers: labels such as “control,” “mild,” and “severe” are just as suitable.

A feature space and a response space need not have the same form. A feature may be a real vector while a response is a finite label; alternatively, a feature might itself be a category and the response a real measurement. The spaces describe the possible values, while their sigma-algebras say which subsets of those values support probability statements.

Measurability and Probability Laws

Once a random element is measurable, its distribution on the value space is defined by transferring probabilities of value sets back to events in the outcome space. This construction is often called the pushforward distribution.

Proof. First, \(X^{-1}(S)=\Omega\), so \(P_X(S)=P(\Omega)=1\). Also \(X^{-1}(\varnothing)=\varnothing\), giving \(P_X(\varnothing)=0\). Let \(A_1,A_2,\ldots\) be pairwise disjoint members of \(\mathcal S\). Their preimages are pairwise disjoint: if an outcome belonged to \(X^{-1}(A_i)\cap X^{-1}(A_j)\), its image would belong to \(A_i\cap A_j=\varnothing\). Preimages also commute with unions, so

The final equality follows from countable additivity of \(P\), since the preimages are disjoint events in \(\mathcal F\). Thus \(P_X\) is nonnegative, has total mass one, and is countably additive on \(\mathcal S\); it is a probability measure. \(\square\)

A Useful Test for Real-Valued Random Variables

For real-valued \(X\), a commonly used way to check measurability is to examine the events \(\{X\le t\}\). The following result explains why checking these threshold events is enough, provided they are measurable for every real threshold.

Proof. If \(X\) is measurable, then \((-\infty,t]\) is Borel for every \(t\), so \(X^{-1}((-\infty,t])=\{\omega:X(\omega)\le t\}\) belongs to \(\mathcal F\).

Conversely, suppose every such threshold event belongs to \(\mathcal F\). Define \(\mathcal C=\{B\in\mathcal B(\mathbb R):X^{-1}(B)\in\mathcal F\}\). More generally, the same preimage condition defines a sigma-algebra on all subsets of \(\mathbb R\): preimages preserve complements and countable unions, and \(\mathcal F\) is a sigma-algebra. By assumption, this sigma-algebra contains every ray \((-\infty,t]\). These rays generate \(\mathcal B(\mathbb R)\): for example, \((-\infty,b)=\bigcup_{q\in\mathbb Q,\ q<b}(-\infty,q]\), and \((a,b)=(-\infty,b)\cap((-\infty,a])^c\). Open intervals generate the open sets using the countable rational interval basis, and open sets generate the Borel sigma-algebra. Hence the preimage of every Borel set belongs to \(\mathcal F\), which is exactly measurability of \(X\). \(\square\)

This criterion is useful because it turns a requirement over all Borel sets into a check of a familiar family of threshold events. It is a criterion for measurability, not a claim that every event of interest must itself be a threshold event.

Why the Choice of Sigma-Algebra Matters

The power set is a natural choice for a finite label space because it allows every group of labels to be discussed. On more general sets, a sigma-algebra may deliberately include fewer subsets. If a proposed value space has a limited sigma-algebra, probability statements about omitted subsets are not defined through that space.

Common Pitfall

Wrong reasoning: “The value space is a set of possible responses, so any subset of responses can automatically be assigned a probability.” A set of possible values alone does not supply a sigma-algebra, and even a specified sigma-algebra may omit some subsets.

Correction: State the measurable space for the feature and response values, and check that the maps from outcomes into those spaces are measurable. Then probability statements about measurable value sets are defined by taking their preimages. For real-valued measurements, use the Borel sigma-algebra; for a finite label set, the power set is the standard choice.

Feature and response spaces make the kinds of value-based questions in a statistical experiment precise. The distribution of a feature or response is a probability measure on its specified measurable space. More complicated objects, such as an entire observed dataset, require a measurable structure on the combined values; that construction is a separate step.