Why Model Labels Matter
A regression data model specifies which distributions could have generated the observed features and responses. The label parametric, semiparametric, or nonparametric describes how strongly the model restricts those distributions; it does not, by itself, say which prediction target is important or which estimator should be used. To understand an assumption, ask what part of the data law it constrains: the conditional response distribution, the covariate distribution, or both.
The distinctions are about the size and structure of a model class. A finite-dimensional formula can rule out many possible data laws. A semiparametric model combines a finite-dimensional component with a flexible, often infinite-dimensional component. A nonparametric model leaves a broad class of laws available, though it can still impose important conditions such as support or moment restrictions.
These labels need a stated data object. A conditional response model can be parametric even when the distribution of random features is unrestricted. In that case, the joint model for \((X,Y)\) is generally semiparametric, not parametric: it combines a restricted conditional law with a flexible covariate law. Conversely, in fixed-design regression, the feature values are treated as fixed, so a finite-dimensional model for the stochastic responses is a parametric model for the experiment.
Parametric Regression: A Formula Restricts the Response Law
Consider fixed feature values \(x_1,\ldots,x_n\), with independent responses satisfying \(Y_i\sim N(\alpha+\beta x_i,\sigma^2)\), where \(\alpha,\beta\in\mathbb{R}\) and \(\sigma>0\). The family is indexed by three real parameters. It constrains the response distributions to be normal, makes their means affine in the feature, and requires a common variance. Many other response laws are excluded.
Worked Example: Reading the Restrictions in a Gaussian Regression Model
Suppose the fixed feature values are \(0,1,2\), and the model is \(Y_i\sim N(1+2x_i,4)\), independently. The parameter values are \(\alpha=1\), \(\beta=2\), and \(\sigma^2=4\).
At \(x=0\), the response distribution is \(N(1,4)\); at \(x=1\), it is \(N(3,4)\); and at \(x=2\), it is \(N(5,4)\). Thus the mean changes by \(2\) for each one-unit increase in \(x\), while the variance remains \(4\). For example, at \(x=2\), \[ P(Y\leq 5\mid X=2)=\frac{1}{2}, \] because \(5\) is the mean of the normal distribution \(N(5,4)\).
The three parameters determine the whole family of conditional response laws. A distribution with a nonnormal conditional shape, a non-affine conditional mean, or a variance that changes with \(x\) does not belong to this model. Those exclusions are modeling assumptions, not consequences of having observed a regression data set.
A model can be parametric in one component while leaving another unrestricted. For random-design data, suppose \(X\) has an arbitrary distribution \(Q\), and \(Y\mid X=x\) follows the Gaussian regression model above. The conditional response law is described by \((\alpha,\beta,\sigma)\), but \(Q\) can be an arbitrary distribution. The joint law is then naturally described as a semiparametric model: a finite-dimensional conditional component and an infinite-dimensional nuisance component \(Q\).
Semiparametric Regression: A Finite Target With a Flexible Law
A common semiparametric structure constrains a feature of the conditional response distribution, such as its mean, while leaving the rest of that distribution flexible. For example, let \(X\) be real-valued and suppose \[ E[Y\mid X]=\alpha+\beta X \] for some finite-dimensional \((\alpha,\beta)\). Apart from existence of the conditional mean, this condition does not specify the shape, variance, or other features of the conditional distribution of \(Y\mid X\).
The last distinction is important. If \(X\) takes only three possible values and its probabilities are unrestricted positive numbers \(p_0,p_1,p_2\) summing to one, then the covariate distribution has two free parameters: \(p_2=1-p_0-p_1\). That is a finite-dimensional nuisance component, not an infinite-dimensional one. An arbitrary distribution over a continuous feature space, by contrast, is generally infinite-dimensional. Conditional response distributions over a continuum of feature values can also contribute an infinite-dimensional nuisance component.
Worked Example: A Linear Conditional Mean Without a Normal Law
Let \(X\) be uniform on \([0,1]\), and set \(\alpha=1\) and \(\beta=2\). Given \(X=x\), let \(Y\) equal \(1+2x-1\) or \(1+2x+1\), each with probability \(1/2\). The conditional mean is \[ E[Y\mid X=x] =\tfrac12(1+2x-1)+\tfrac12(1+2x+1) =1+2x. \] So the linear conditional-mean restriction holds.
At each \(x\), this particular conditional distribution has two equally likely values, rather than a normal shape. More generally, the model permits many conditional distributions as long as their means equal \(1+2x\). For example, the two-point construction could be replaced by other distributions centered at the same mean. The finite-dimensional mean parameter therefore does not determine the full conditional law.
Here the covariate distribution is uniform, but the semiparametric model could allow \(Q\) to vary over a broad class of distributions. The example's chosen uniform law is one instance; it does not turn the unrestricted covariate component into a finite-dimensional one.
Nonparametric Models: Broad Classes, Not No Assumptions
A nonparametric model can allow the joint distribution of \((X,Y)\), or the conditional distribution of \(Y\mid X\), to vary without imposing a fixed finite-dimensional formula. The class still needs to be specified. For example, one may require that \(Y\) have a finite mean, or that the support of \(X\) lie in a particular set. Such requirements restrict the allowed laws, but they do not generally reduce the model to a fixed finite-dimensional parameter.
Worked Example: An Unrestricted Conditional Probability Function
Let \(X\) be uniform on \([0,1]\), let \(Y\) be binary, and specify \(P(Y=1\mid X=x)=x^2\). The conditional probability at \(x=0.5\) is \(0.25\). The marginal probability of a positive response is \[ P(Y=1)=\int_0^1 x^2\,dx =\left[\frac{x^3}{3}\right]_0^1 =\frac13. \] Likewise, \[ P(X\leq 0.5,\ Y=1)=\int_0^{0.5}x^2\,dx =\left[\frac{x^3}{3}\right]_0^{0.5} =\frac{(0.5)^3}{3} =\frac{1}{24}. \]
The function \(x^2\) is one possible conditional probability function. A nonparametric model for binary responses can allow an arbitrary measurable function \(q(x)\) taking values in \([0,1]\), rather than specifying that it must be \(x^2\) or belong to another fixed finite-dimensional family. Choosing \(x^2\) for this calculation illustrates a law in the class; it is not a restriction defining the whole class.
“Nonparametric” does not mean that all distributions are automatically allowed, nor does it mean that a model has no structure. A model may be nonparametric in its response law but assume a particular feature support, or restrict attention to distributions with finite moments. The useful question remains: which parts of the data law are fixed by the model and which can vary freely?
How the Model Classes Relate
The relationship among model classes can be made precise for conditional response distributions. Let \(\mathcal{M}_0\) be the class of all conditional distributions with finite conditional first moments on the feature support. Let \(\mathcal{M}_1\) contain those distributions satisfying \(E[Y\mid X=x]=\alpha+\beta x\) for some \(\alpha,\beta\). Let \(\mathcal{M}_2\) contain those satisfying \(Y\mid X=x\sim N(\alpha+\beta x,\sigma^2)\) for some \(\alpha,\beta\) and \(\sigma>0\). Compare these classes for the same feature support and, if the feature is random, the same \(Q\).
Proof. If a conditional distribution belongs to \(\mathcal{M}_2\), then its conditional mean is \(\alpha+\beta x\), since the mean of \(N(\alpha+\beta x,\sigma^2)\) is \(\alpha+\beta x\). It therefore belongs to \(\mathcal{M}_1\). Every member of \(\mathcal{M}_1\) has finite conditional first moments by the definition of that class, so \(\mathcal{M}_1\subseteq\mathcal{M}_0\).
To show the first inclusion is strict, take \(X\) with support \(\{0,1,2\}\), and let \(Y\mid X=x\) be Bernoulli with probabilities \(0.2,0.5,0.8\) at those three feature values, respectively. Its conditional mean equals \(0.2+0.3x\), so it is in \(\mathcal{M}_1\). It cannot be in \(\mathcal{M}_2\), because each conditional distribution is supported on \(\{0,1\}\), whereas a normal distribution with \(\sigma>0\) is not.
To show the second inclusion is strict, use the same feature support and let \(Y\mid X=x\sim N(x^2,1)\). This law has finite conditional first moments, so it belongs to \(\mathcal{M}_0\). It cannot belong to \(\mathcal{M}_1\): an affine function matching the conditional means \(0\) at \(x=0\) and \(1\) at \(x=1\) must equal \(x\), which gives \(2\) at \(x=2\), not the required \(4\). Thus both inclusions are strict. \(\square\)
A Concrete Parametric Restriction in Logistic Regression
A finite-dimensional conditional model can impose a checkable relation among conditional probabilities. Suppose \(X\in\{0,1,2\}\), \(Y\) is binary, and \[ P(Y=1\mid X=x)=\frac{\exp(\alpha+\beta x)}{1+\exp(\alpha+\beta x)}. \] Writing \(q_x=P(Y=1\mid X=x)\), the model requires each \(q_x\) to lie strictly between zero and one.
Proof. Under the stated model, \(\operatorname{logit}(q_x)=\alpha+\beta x\). Substituting \(x=0,1,2\) gives \(\operatorname{logit}(q_0)=\alpha\), \(\operatorname{logit}(q_1)=\alpha+\beta\), and \(\operatorname{logit}(q_2)=\alpha+2\beta\). Hence the middle value is the average of the other two. Conversely, if the displayed relation holds, choose \(\alpha=\operatorname{logit}(q_0)\) and \(\beta=\operatorname{logit}(q_1)-\operatorname{logit}(q_0)\). The relation implies \(\operatorname{logit}(q_2)=\alpha+2\beta\), so all three probabilities have the required logistic form. \(\square\)
Worked Example: Checking a Logistic Model Constraint
For \(q_0=0.2,\ q_1=0.5,\ q_2=0.8\), the logits are \(-\log 4,\ 0,\ \log 4\). Their middle value is the average of the endpoints: \[ \frac{-\log 4+\log 4}{2}=0. \] The probabilities satisfy the restriction. In fact, \(\alpha=-\log 4\) and \(\beta=\log 4\) produce them.
Now change only the middle probability to \(q_1=0.6\). Its logit is \(\log(3/2)\), which is positive, while the required midpoint remains \(0\). Therefore \((0.2,0.6,0.8)\) cannot come from this logistic model. It is allowed by a model that places no such finite-dimensional restriction on the three conditional probabilities.
Common Pitfall: Confusing “Nuisance” With “Infinite-Dimensional”
A common mistake is to call every nuisance component infinite-dimensional. “Nuisance” means that a component is not the parameter of primary interest; it says nothing about its dimension. For example, if the feature takes three possible values with probabilities \(p_0,p_1,p_2>0\), then \(p_0+p_1+p_2=1\), leaving two free probabilities. These are finite-dimensional nuisance parameters. An arbitrary distribution over a continuum of feature values is generally infinite-dimensional.
Another mistake is to treat a parametric conditional model as a parametric model for the entire random-design law. If the covariate distribution is left unrestricted, the joint law retains that flexible component. State which law is being modeled, then identify each component and the restrictions placed on it.
Check Your Understanding
Use the model structure, not just its name, to answer each question.
- In fixed-design Gaussian regression with unknown intercept, slope, and common variance, name the finite-dimensional parameters and two assumptions imposed on the responses.
- A random-design model specifies a logistic conditional response law but leaves the feature distribution arbitrary. Why need the joint model not be called purely parametric?
- If a feature has three possible values with unrestricted positive probabilities summing to one, how many free probabilities describe its distribution?
- Give one way a conditional-mean model can be semiparametric even when its mean is determined by two coefficients.
- For a logistic regression on feature values \(0,1,2\), state the relation among the three conditional logits that the model requires.