Why Diagnose the Experiment Before Naming the Target?
A prediction target is not determined by the word “regression” or “classification.” It is determined by a statistical experiment: what is sampled, what is held fixed, and what probability law describes the observations. For example, a study that samples new covariates and responses from a population poses a different target question from a study that fixes covariate values in advance and repeatedly samples responses at those values.
A careful diagnosis separates four items: the sampling protocol, the random quantities, the assumptions about their joint distribution, and the target those assumptions define. This prevents a common mismatch: treating a function estimated from observed covariate values as if the experiment automatically defined a population target at every possible covariate value.
For clarity, let \((\mathcal X,\mathcal A)\) be the feature space and \((\mathcal Y,\mathcal B)\) the response space. In the usual real-valued and finite-class examples, these spaces carry their Borel and discrete sigma-algebras, respectively. A measurable prediction function maps feature values into a suitable prediction space. The dataset itself is also a random element: for \(n\) observed pairs, it takes values in the product space \((\mathcal X\times\mathcal Y)^n\) with its product sigma-algebra.
A Diagnostic Sequence
Work through the following questions in order. The answers determine what is random and what the experiment can identify.
Say whether covariates are sampled from a population, fixed by design, or generated by some other stated mechanism. Also say whether observations are independent or dependent if that assumption is part of the setup.
Separate random covariates, random responses, and any auxiliary randomization used by a learning procedure. Do not describe a fixed design value as random merely because it appears in a data table.
Specify a joint law, or a covariate law together with conditional response laws. State independence, support, and moment conditions needed for the target you name.
Identify the feature, response, and dataset spaces. This makes the domain of a prediction function and the random object under study precise.
For regression, specify the conditional mean when it exists; for classification, specify the conditional class probabilities. Say whether the target is population-based, defined at fixed design points, or only identified almost surely.
For independent and identically distributed (IID) pairs, the covariate distribution and the conditional response law together specify the joint distribution. In regression, if \(E[|Y|]<\infty\), the conditional mean is a prediction target. In classification with labels \(1,\ldots,K\), the conditional class probabilities are \(P(Y=k\mid X=x)\), for each class \(k\). These are targets describing the response law; selecting a final class label is a separate decision.
Worked Regression Diagnosis
Worked Example: IID Regression on a Finite Covariate Space
Suppose each study participant contributes a covariate \(X\in\{0,1,2\}\) and a real-valued response \(Y\). Participants are sampled independently from the same population. The covariate probabilities are \(P(X=0)=0.2\), \(P(X=1)=0.5\), and \(P(X=2)=0.3\). Conditional on \(X=x\), assume \(Y\mid X=x\sim N(1+2x,1+x)\), where the second normal parameter is the variance. Diagnose the experiment and state its regression target.
Sampling and random quantities. The observations \((X_i,Y_i)\), \(i=1,\ldots,n\), are IID copies of \((X,Y)\). Both \(X_i\) and \(Y_i\) are random. The observed dataset is a random element of \((\{0,1,2\}\times\mathbb R)^n\), with the discrete sigma-algebra on the finite set and the Borel sigma-algebra on \(\mathbb R\).
Distributional assumptions. The marginal law of \(X\) is the three-point distribution given above. At each supported covariate value, \(Y\) is conditionally normal with the stated mean and variance. In particular, the response has a finite conditional mean at each point in the support.
Target calculation. The regression target is \(m(x)=E[Y\mid X=x]\), at covariate values with positive probability. From the conditional normal model, \(m(x)=1+2x\). Thus \(m(0)=1\), \(m(1)=3\), and \(m(2)=5\). For example, at \(x=2\), the response distribution is \(N(5,3)\), whose mean is 5 and variance is 3.
The experiment defines this population target on the support \(\{0,1,2\}\), or equivalently as a target up to \(P_X\)-almost-sure equality. The formula \(1+2x\) could be evaluated at other real values, but the stated covariate law assigns those values probability zero. The formula alone does not turn the experiment into evidence about a population target outside its covariate support.
Worked Classification Diagnosis
Worked Example: Class Probabilities Under IID Sampling
A diagnostic study samples patients independently from a population. The feature \(X\) records one of two test profiles, \(A\) or \(B\), with \(P(X=A)=0.4\) and \(P(X=B)=0.6\). The response \(Y\in\{0,1\}\) records whether a condition is present. Assume \(P(Y=1\mid X=A)=0.2\) and \(P(Y=1\mid X=B)=0.75\). Identify the experiment and its classification target.
Spaces and random quantities. Each \(X_i\) is a random element of the finite space \(\{A,B\}\), and each \(Y_i\) is a random element of \(\{0,1\}\); both spaces carry their discrete sigma-algebras. The dataset takes values in \((\{A,B\}\times\{0,1\})^n\). The protocol makes the pairs IID, so both the profile and the condition status vary across repeated samples.
Distributional assumptions. The marginal probabilities for \(X\) and the two conditional response probabilities specify the joint law. For example, \(P(X=A,Y=1)=P(X=A)P(Y=1\mid X=A)=0.4(0.2)=0.08\). The other joint probabilities are \(P(A,0)=0.4(0.8)=0.32\), \(P(B,1)=0.6(0.75)=0.45\), and \(P(B,0)=0.6(0.25)=0.15\). They sum to \(0.08+0.32+0.45+0.15=1\), as required.
Target. The class-probability target is \(\eta(x)=P(Y=1\mid X=x)\), so \(\eta(A)=0.2\) and \(\eta(B)=0.75\). Since there are two classes, the full conditional probability vectors are \((0.8,0.2)\) at \(A\) and \((0.25,0.75)\) at \(B\), ordered as classes 0 and 1. The target concerns conditional probabilities, not a claim that every patient with profile \(A\) or \(B\) has a fixed outcome.
Worked Fixed-Design Diagnosis
Worked Example: Regression at Prespecified Covariate Values
A laboratory chooses three settings in advance, \(x_1=0\), \(x_2=1\), and \(x_3=2\), then records one response at each setting. The settings are fixed, and the responses are assumed independent with \(Y_i\sim N(1+2x_i,1)\). State what is random, what the design implies, and what target is specified.
Sampling and spaces. The covariates \(x_1,x_2,x_3\) are constants set by the design, not random draws from a covariate distribution. The response vector \((Y_1,Y_2,Y_3)\) is random and takes values in \(\mathbb R^3\) with its product Borel sigma-algebra. Its stated joint law is the product of the three normal response laws because independence is assumed.
Target at the design points. The conditional mean at each setting is \(E[Y_i]=1+2x_i\). Substituting the three fixed values gives \(E[Y_1]=1\), \(E[Y_2]=3\), and \(E[Y_3]=5\). Thus the experiment specifies the mean vector \((1,3,5)\) at the selected design points.
Unlike the IID examples, this setup does not specify a marginal distribution \(P_X\): no covariates were sampled from a population. The response assumptions give means at the chosen settings, but do not by themselves define a population-average prediction target over new covariates. If the laboratory instead repeated measurements at each setting, the sampling protocol and the response vector would need to be described with those replicates included.
Two Useful Derivations
The following facts make the connection between a sampling description and a prediction target explicit. In each, the assumptions are part of the conclusion: changing the sampling protocol can change the target being discussed.
Proof. The defining properties of a probability kernel imply that \(h\) is measurable (first for nonnegative measurable \(g\), then for integrable \(g\) by positive and negative parts). Let \(A\in\mathcal A\). The conditional distribution property gives the integration identity
Integrability of \(g(Y)\) ensures these expectations exist; the identity for signed \(g\) follows by applying the nonnegative identity to its positive and negative parts. Since the equality holds for every measurable \(A\), \(h(X)\) satisfies the defining integral property of \(E[g(Y)\mid X]\). Hence \(h\) is a version of that conditional expectation, which proves the proposition. \(\square\)
For regression, take \(g(y)=y\) when \(E[|Y|]<\infty\). For classification, take \(g(y)=\mathbf 1_{\{y=k\}}\); the resulting conditional expectation is the class probability for class \(k\). The derivation explains why a joint law, through its conditional response distribution, determines these targets.
Derivation. Independence means that for measurable sets \(B_1,\ldots,B_n\subseteq\mathbb R\), \(P(Y_1\in B_1,\ldots,Y_n\in B_n)=\prod_{i=1}^n Q_i(B_i)\). These probabilities on measurable rectangles determine the product probability law on the product sigma-algebra, so the response vector has law \(\prod_i Q_i\). Each coordinate mean is its integral under the corresponding marginal law \(Q_i\), giving the displayed mean vector. The covariates are constants and therefore contribute no marginal random distribution \(P_X\) to this experiment. \(\square\)
Common Mistake: Treating the Design as a Population
A frequent error is to see several covariate values in a dataset and conclude that they were sampled from a population distribution. The reasoning is: “The data contain \(x_1,\ldots,x_n\), so these values are observations of random \(X\), and the experiment defines a population target over \(X\).” That conclusion does not follow. In the fixed-design example, the \(x_i\) were chosen in advance; only the responses vary under repeated sampling.
The correction is to report the protocol before naming the target. Under IID random-design sampling, the law includes \(P_X\) and a conditional response law, so the conditional target is relevant on the support of \(P_X\), up to almost-sure equality. Under fixed design, state the target at the specified design points unless additional assumptions define a broader function or covariate population. A formula that happens to extend beyond those points is not, by itself, a distributional assumption about new covariates.
Check Your Understanding
For each question, explain how the sampling protocol supports your answer.
- In an IID regression study, what components of the law are needed to specify the joint distribution of a feature and response?
- A study fixes the same covariate value for every participant and samples a response for each. Which quantity is fixed, and what additional information would be needed to define a population covariate distribution?
- In binary classification, how can the conditional class probability be written as a conditional expectation?
- Why does a conditional target specified under a random-design law generally need only be identified almost surely under the covariate distribution?
- A model gives a conditional normal response distribution at three prespecified covariate values. What target can be reported directly, and why does this not alone determine a target over all real covariate values?