Statistical Experiments for Supervised Learning

What Is Random, What Is Unknown, and What Is Learned?

A supervised-learning analysis begins with a distinction that is easy to blur: the data we happen to observe are not the same thing as the probability law that could have generated them, and neither is the prediction rule we construct from them. Keeping these objects separate clarifies what a learning method can use and what its output means.

For example, a dataset might contain records of covariates and responses. Before collection, that dataset is uncertain and is described by a probability law. Some unknown feature of that law—perhaps a probability or another parameter—is the object we hope to learn about. A procedure then uses the realized dataset to produce a prediction rule. The rule depends on the data, but it is not itself the data-generating law.

Definition: A statistical experiment is a triple $(\mathcal{D},\mathcal{A},\{P_\theta:\theta\in\Theta\})$. Here $\mathcal{D}$ is the space of possible datasets, $\mathcal{A}$ is a sigma-algebra of events in that space, $\Theta$ is a parameter space, and each $P_\theta$ is a probability law on $(\mathcal{D},\mathcal{A})$. The unknown $\theta$ indexes the data-generating law. The observed dataset is a realized value $d\in\mathcal{D}$ of a random dataset $D$.

The parameter $\theta$ is not necessarily the entire law $P_\theta$. It may describe a feature of that law, and different parameter values may even correspond to the same law. The family $\{P_\theta\}$ states which data laws are being considered; it does not assert that any one of them is known to be the true law. In an analysis, one member of the family is taken to govern the data, but its index is unknown.

From a Dataset to a Prediction Rule

Let $\mathcal{X}$ denote an input space and $\mathcal{Y}$ a response or action space. A prediction rule is a function $g:\mathcal{X}\to\mathcal{Y}$. A deterministic learning procedure is a map $A$ that takes a dataset $d$ and returns a prediction rule $A(d)$. Thus, when the observed dataset is $d$, the constructed rule is $g_d=A(d)$.

Key distinction: Before observing the data, $D$ is random, so $A(D)$ is a random prediction rule. After observing $D=d$, the output $g_d=A(d)$ is fixed, provided the procedure itself has no additional randomness. The data-generating law $P_\theta$ is a probability distribution on possible datasets; it is neither the realized dataset nor the rule produced by the learning procedure.

To discuss the probability distribution of a learned rule, the output must also be a measurable object. One simple setting is a finite set $\mathcal{D}$ of possible datasets and a finite collection $\mathcal{G}$ of candidate rules. Then every map $A:\mathcal{D}\to\mathcal{G}$ is measurable. More generally, equip $\mathcal{G}$ with a sigma-algebra and require $A$ to be measurable. This condition ensures that events about the output rule correspond to measurable events about the data.

Worked Example: A Dataset, Its Law, and a Rule

Suppose one record consists of an input $u\in\{\text{red},\text{blue}\}$ and a binary response $v\in\{0,1\}$. The dataset space is $\mathcal{D}=\{(\text{red},0),(\text{red},1),(\text{blue},0),(\text{blue},1)\}$. For an unknown $\theta\in[0,1]$, let the data law assign probabilities $P_\theta((\text{red},1))=\theta/2$, $P_\theta((\text{red},0))=(1-\theta)/2$, $P_\theta((\text{blue},1))=\theta/2$, and $P_\theta((\text{blue},0))=(1-\theta)/2$.

These are valid probabilities: their sum is $\theta/2+(1-\theta)/2+\theta/2+(1-\theta)/2=1$. The unknown feature is $\theta$; the data-generating law is $P_\theta$; and the observed dataset is one of the four records. For instance, if $\theta=0.6$, the probability of observing $(\text{blue},1)$ is $0.6/2=0.3$. Observing that record does not mean the law is the record, nor does it reveal the value of $\theta$ with certainty.

Consider a procedure that returns a constant prediction equal to the observed response: $A((u,v))(x)=v$ for every input $x$. If the observed record is $(\text{blue},1)$, the constructed rule predicts 1 for all inputs. If the record is $(\text{red},0)$, it predicts 0 for all inputs. The procedure maps a dataset to a rule; the law determines how likely each dataset, and therefore each output, is to be.

The Distribution of a Learned Rule

A learning procedure transforms the randomness in the dataset into randomness in its output. The precise statement is the pushforward law: probabilities of output events are obtained by taking the probability of the corresponding datasets under the original data law.

Theorem: Suppose $D$ has law $P_\theta$ on $(\mathcal{D},\mathcal{A})$, $(\mathcal{G},\mathcal{B})$ is a measurable space of prediction rules, and $A:\mathcal{D}\to\mathcal{G}$ is measurable. Then the law of $A(D)$, denoted $Q_\theta$, satisfies $$ Q_\theta(B)=P_\theta\bigl(\{d\in\mathcal{D}:A(d)\in B\}\bigr) =P_\theta(A^{-1}(B)),\qquad B\in\mathcal{B}. $$

Proof. By definition, $Q_\theta(B)=P_\theta(A(D)\in B)$. The event $\{A(D)\in B\}$ is exactly $\{D\in A^{-1}(B)\}$, where $A^{-1}(B)=\{d:A(d)\in B\}$. Measurability of $A$ makes this a valid event in $\mathcal{A}$. Since $D$ has law $P_\theta$, its probability is $P_\theta(A^{-1}(B))$. This holds for every $B\in\mathcal{B}$, so it gives the output law. $\square$

Worked Example: Computing the Law of the Output Rule

Suppose a dataset $D$ takes three possible values $d_0,d_1,d_2$, with $P_\theta(D=d_0)=0.2$, $P_\theta(D=d_1)=0.5$, and $P_\theta(D=d_2)=0.3$. A procedure maps $d_0$ and $d_1$ to rule $g_0$, and maps $d_2$ to a different rule $g_1$.

The event that the output is $g_0$ corresponds to the two datasets $d_0,d_1$. Therefore $P_\theta(A(D)=g_0)=0.2+0.5=0.7$. The event that the output is $g_1$ corresponds only to $d_2$, so $P_\theta(A(D)=g_1)=0.3$. These probabilities sum to $0.7+0.3=1$. The procedure has discarded which of $d_0$ and $d_1$ occurred, because both produce the same rule. The output law records this mapping, rather than being identical to the dataset law.

When Different Unknown Values Cannot Be Distinguished

An experiment can distinguish parameter values only through differences in the data laws they specify. If two parameter values induce exactly the same law on the dataset space, no event defined from the observed dataset can have different probabilities under those values. A deterministic learning procedure cannot create a distinction that was absent from the data law.

Proposition: If $P_{\theta_0}=P_{\theta_1}$ on $(\mathcal{D},\mathcal{A})$, then every measurable deterministic procedure $A$ has the same output law under $\theta_0$ and $\theta_1$.

Proof. Let $B$ be any measurable set of prediction rules. By the pushforward formula, $P_{\theta_j}(A(D)\in B)=P_{\theta_j}(A^{-1}(B))$ for $j=0,1$. The set $A^{-1}(B)$ is measurable. Since $P_{\theta_0}=P_{\theta_1}$, the two probabilities assigned to this set are equal. This holds for every $B$, so the output laws are equal. $\square$

Worked Example: Two Parameter Values with the Same Data Law

Let $\Theta=\{0,1\}$ and $\mathcal{D}=\{a,b\}$. Suppose both parameter values give $P_\theta(D=a)=0.4$ and $P_\theta(D=b)=0.6$. The two labels $\theta=0$ and $\theta=1$ are different, but the specified data laws are identical.

For any event $C\subseteq\{a,b\}$, its probability is the same under both parameter values. For example, $P_0(D=a)=0.4=P_1(D=a)$, and $P_0(D\in\{a,b\})=1=P_1(D\in\{a,b\})$. If a procedure maps $a$ to $g_a$ and $b$ to $g_b$, its output probabilities are $0.4$ for $g_a$ and $0.6$ for $g_b$ under either value. No method using only this dataset can infer which label generated it from a difference in these laws, because there is no such difference.

A Common Mistake and Why the Distinction Matters

A common mistake is to say, “The model is the dataset,” or, conversely, “The learned rule is the data-generating distribution.” Both statements conflate objects with different roles. The dataset is a realized outcome; the law assigns probabilities to possible outcomes; and the prediction rule is computed from the realized outcome. A procedure may produce a rule that is useful for prediction without recovering the full data law or identifying every feature that indexes it.

It is also important not to confuse a random output with a fixed one. Before the dataset is observed, the procedure’s output can vary across possible datasets, and its distribution is induced by the data law. Once a particular dataset is observed and a deterministic procedure is applied, the resulting rule is fixed. This distinction provides the foundation for later discussions of sampling schemes, prediction targets, and the assumptions that restrict possible data laws.

Key takeaway: A supervised-learning experiment specifies a family of laws for possible datasets. The unknown parameter indexes those laws, the observed dataset is one realization, and a learning procedure maps that realization to a prediction rule.

Check Your Understanding

For each question, keep the unknown parameter, data law, realized data, and learned rule distinct.

In a statistical experiment, what are the roles of the sample space, the probability law, and the parameter?
Before data are observed, why is the output of a deterministic learning procedure generally random?
If a procedure maps two different datasets to the same prediction rule, how is the probability of that rule calculated?
Suppose two parameter values induce identical laws on the dataset space. Can a deterministic learning procedure have different output laws under those values? Explain.
Give one example of a statement that incorrectly identifies the observed dataset with either the data-generating law or the prediction rule, and correct it.