Model-Based Assumptions as Restrictions on Data Laws

Introduction

A statistical model is not only a formula for a fitted predictor. It is also a claim about which probability laws could have generated the data. A regression equation such as “the conditional mean is linear” rules out some data-generating distributions, but it does not, by itself, specify all features of the response distribution. A noise assumption can impose additional restrictions.

Making these restrictions explicit helps distinguish assumptions that may sound similar. Two models can have the same regression function and very different conditional response distributions. Conversely, a proposed equation and noise law may be incompatible with the possible values of the response. The basic object throughout is the law of the observed data, including the covariates as well as the responses.

Data Laws and Model Restrictions

For a single observation, write $X$ for the covariate and $Y$ for the response. Under random-design sampling, their joint distribution determines both the covariate distribution and the response distribution conditional on the covariate. In ordinary real-valued settings, it can be described by a marginal law $P_X$ and a conditional distribution $K_x$ for $Y$ given $X=x$. For measurable sets $A$ and $B$, the joint law satisfies

P(X\in A,\;Y\in B)=\int_A K_x(B)\,P_X(dx).

This equation shows where a modeling assumption can act. A model may restrict $P_X$, the conditional distributions $K_x$, or both. Often the covariate distribution is left unrestricted and assumptions are placed on the response given the covariate. The model is then a collection of allowed joint laws, rather than one particular law.

Definition: A model for the data law is a specified collection of probability distributions that are allowed to generate the observations. A model-based assumption is a restriction requiring the true data law to belong to that collection. A regression equation is one possible restriction on the conditional distribution of $Y$ given $X$; it is not generally a complete specification of that distribution.

For example, suppose a model requires

E[Y\mid X=x]=m_\theta(x),

where $m_\theta$ is a specified function indexed by a parameter $\theta$. This constrains the conditional mean wherever the covariate distribution puts positive probability, or more generally for $P_X$-almost every $x$. It does not, on its own, determine conditional variances, tail probabilities, or the shape of the conditional distributions. Different laws for $Y\mid X=x$ can have the same mean.

The phrase “for $P_X$-almost every $x$” matters: conditional distributions at covariate values that occur with probability zero do not affect the joint data law. A model can also state a formula for all $x$, but data from that law only directly constrain the conditional distribution on the covariate values covered by $P_X$.

Worked Example: A Linear Mean Restriction for a Binary Response

Let $X$ take values 0 and 1, with $P(X=0)=0.4$ and $P(X=1)=0.6$. Let $Y$ be binary, and suppose the model says $P(Y=1\mid X=x)=0.2+0.6x$. Since a binary response has conditional mean equal to its probability of being 1, this is also the regression equation $E[Y\mid X=x]=0.2+0.6x$.

At $x=0$, $P(Y=1\mid X=0)=0.2$, so $P(Y=0\mid X=0)=0.8$. At $x=1$, the corresponding probabilities are $0.8$ and $0.2$. Multiplying each conditional probability by the covariate probability gives the joint law:

\begin{aligned} P(X=0,Y=0)&=0.4(0.8)=0.32, & P(X=0,Y=1)&=0.4(0.2)=0.08,\\ P(X=1,Y=0)&=0.6(0.2)=0.12, & P(X=1,Y=1)&=0.6(0.8)=0.48. \end{aligned}

The four probabilities sum to $0.32+0.08+0.12+0.48=1$, so they define a joint distribution. The model restricts the conditional response probabilities at the two covariate values that occur. It makes no claim about a conditional probability at $x=2$, because $X$ never takes that value under this design.

From a Regression Equation to Moment Restrictions

A conditional-mean equation can also be expressed through expectations involving the residual $Y-m_\theta(X)$. This gives a useful way to see exactly what the equation requires of the data law.

Theorem: Suppose $X$ and $Y$ are real-valued, $E[|Y|]<\infty$, and $m(X)$ is integrable. Then $E[Y\mid X]=m(X)$ almost surely if and only if $$ E\!\left[(Y-m(X))g(X)\right]=0 $$ for every bounded measurable function $g$.

Proof. First suppose $E[Y\mid X]=m(X)$ almost surely. For bounded measurable $g$, the defining property of conditional expectation gives

E[(Y-m(X))g(X)] =E\!\left[E[Y-m(X)\mid X]\,g(X)\right] =0.

Conversely, define $H=E[Y-m(X)\mid X]$. This is integrable and measurable with respect to $X$. The assumed moment equations imply $E[H g(X)]=0$ for every bounded measurable $g$. In particular, choose $g(X)=\mathbf{1}_{\{H>0\}}$. Then $E[H\mathbf{1}_{\{H>0\}}]=0$, which implies $H$ cannot be positive on a set of positive probability. Choosing $g(X)=\mathbf{1}_{\{H<0\}}$ in the same way shows it cannot be negative on a set of positive probability. Thus $H=0$ almost surely, so $E[Y\mid X]=m(X)$ almost surely. $\square$

This result says that a regression equation imposes an entire family of restrictions, one for each bounded function of $X$. Taking $g(X)=1$ gives the overall mean residual restriction $E[Y-m(X)]=0$, but that single equation is generally much weaker than the conditional-mean assumption. A residual can average to zero overall while having positive average for some covariate values and negative average for others.

Worked Example: An Overall Residual Mean Is Not Enough

Let $X$ be equally likely to be 0 or 1, and let $Y=X$. Consider the proposed regression function $m(X)=1/2$. The residual is $Y-m(X)$, which equals $-1/2$ when $X=0$ and $1/2$ when $X=1$. Therefore,

E[Y-m(X)]=\tfrac12(-\tfrac12)+\tfrac12(\tfrac12)=0.

But the conditional means are $E[Y\mid X=0]=0$ and $E[Y\mid X=1]=1$, neither of which equals $m(X)=1/2$. The full moment restriction detects the failure: with $g(X)=X$,

E[(Y-m(X))X]=\tfrac12(0)(0)+\tfrac12(\tfrac12)(1)=\tfrac14\ne 0.

Thus an overall mean-zero residual does not establish the conditional regression equation. The condition must hold against every bounded function of the covariates.

A Noise Law Adds Distributional Restrictions

A noise assumption can specify more than the conditional mean. For example, suppose $X$ is real-valued, $m$ is a specified regression function, and the conditional law is

Y\mid X=x\sim N(m(x),\sigma^2), \qquad \sigma^2>0,

for $P_X$-almost every $x$. This is a restriction on the full conditional distribution, not merely on its mean. It requires a normal shape and the same conditional variance $\sigma^2$ at the covariate values covered by the model. One can express the same assumption by saying that $Y=m(X)+\varepsilon$, where $\varepsilon$ is independent of $X$ and has distribution $N(0,\sigma^2)$. Under this formulation, both the distribution of the error and its independence from the covariate are assumptions.

Result: If $Y\mid X=x\sim N(m(x),\sigma^2)$ for $P_X$-almost every $x$, then $E[Y\mid X]=m(X)$ and $\operatorname{Var}(Y\mid X)=\sigma^2$ almost surely.

Derivation. For each such $x$, the conditional distribution is a normal distribution with mean $m(x)$ and variance $\sigma^2$. By the mean and variance of the normal distribution, $E[Y\mid X=x]=m(x)$ and $\operatorname{Var}(Y\mid X=x)=\sigma^2$. These identities therefore hold for $P_X$-almost every $x$, which gives the stated conditional moments. The converse does not follow: knowing these two conditional moments does not require the conditional distribution to be normal. $\square$

Worked Example: A Gaussian Conditional Law

Suppose $X$ is equally likely to be 0 or 1 and, given $X=x$, $Y\sim N(1+2x,4)$. At $X=0$, the conditional response is $N(1,4)$; at $X=1$, it is $N(3,4)$. Thus the conditional mean is $1+2x$, and the conditional variance is 4 at both covariate values.

For instance, at $X=1$, the event $Y>5$ has conditional probability

P(Y>5\mid X=1) =P\!\left(\frac{Y-3}{2}>\frac{5-3}{2}\,\middle|\,X=1\right) =P(Z>1)\approx 0.1587,

where $Z\sim N(0,1)$. The specified noise law determines this tail probability, not just the conditional mean and variance. A different conditional distribution with mean 3 and variance 4 could give a different probability for $Y>5$, and would not satisfy this Gaussian model.

The same logic applies when the design is fixed. If the covariates $x_1,\ldots,x_n$ are fixed, a conditional response model restricts the distribution of the response vector given those design values. For example, specifying that each response has a particular conditional normal distribution is not, by itself, a claim that the responses are conditionally independent; independence is an additional restriction on their joint conditional law. A model must state the joint restrictions it needs.

Checking Whether a Model Is Possible

Before using a model, check that its restrictions can hold for the stated response space and covariate distribution. A formula for a conditional mean must be within the range of possible conditional means. For binary $Y$, that range is from 0 to 1. A claimed conditional distribution must also assign probability only to values the response can actually take.

Worked Example: An Impossible Binary-Response Mean Model

Let $X$ take values 0, 1, and 2, each with positive probability, and let $Y$ be binary. Suppose the proposed regression equation is $E[Y\mid X=x]=0.2+0.5x$. At $x=2$, this gives

E[Y\mid X=2]=0.2+0.5(2)=1.2.

But a binary response satisfies $0\leq Y\leq 1$, so its conditional expectation must also satisfy $0\leq E[Y\mid X=2]\leq 1$. Since $P(X=2)>0$, the equation is required at that covariate value and contradicts this bound. No joint distribution for $X$ and $Y$ can satisfy the proposed model.

If $X$ never equalled 2, this calculation would not by itself show that the joint data law is impossible: the conditional-mean restriction is only constrained at values covered by the covariate distribution. The covariate law is therefore part of the check.

Common Pitfall and Why the Distinction Matters

A common mistake is to read a regression equation as a deterministic identity. From $E[Y\mid X]=m(X)$, it does not follow that $Y=m(X)$. The equation specifies a conditional average; individual responses may vary around that average. A noise model may describe that variation, but it adds assumptions that are not contained in the regression equation alone.

A second mistake is to treat a noise distribution as if it only restated the mean. A normal conditional law determines tail probabilities and other distributional features as well as its mean. Using those features in analysis relies on the stronger distributional assumption. Stating the assumptions as restrictions on data laws makes this difference visible: the more detailed the model, the fewer possible laws it permits.

This perspective is useful when interpreting results. A conclusion derived under a restricted model is warranted only to the extent that the data-generating law satisfies the restrictions used. It also helps diagnose incoherent assumptions before fitting a model: check the response range, the covariate values to which equations apply, and whether the specified conditional distributions can coexist with the stated sampling design.

Key takeaway: A model-based assumption defines which data laws are allowed. A regression equation restricts conditional means; a noise law can restrict the full conditional distributions. Neither should be confused with a deterministic response formula, and every restriction must be compatible with the response and covariate spaces.

Check Your Understanding

Use the distinction between conditional-mean and full-distribution restrictions to answer the following.

If $E[Y\mid X]=m(X)$, does this determine $\operatorname{Var}(Y\mid X)$? Explain.
Why does $E[Y-m(X)]=0$ not, by itself, imply $E[Y\mid X]=m(X)$? Give a choice of a function $g(X)$ that can reveal a failure.
If $Y$ is binary and $P(X=2)>0$, can the conditional-mean restriction $E[Y\mid X=x]=1.1$ hold at $x=2$? Why or why not?
What additional restrictions does $Y\mid X=x\sim N(m(x),\sigma^2)$ impose beyond the conditional-mean equation?
In a fixed-design model, why is conditional independence of responses a separate assumption from specifying each response's conditional marginal distribution?