Regression With Prescribed Feature Points
In many regression problems, the feature values are chosen in advance: a laboratory specifies a set of input levels, or a study records outcomes at scheduled measurement times. The responses are still uncertain. Repeating the experiment with the same feature points could produce different response values.
This setup separates two roles. The feature points describe where responses are observed; the responses are random quantities generated at those points. A regression model can describe how the response distribution, or a feature of that distribution such as its mean, changes across the prescribed points. No linear form is required by the setup itself.
For real-valued responses, write \(F_x(t)=P_x(Y\le t)\) for the distribution function of a response generated at feature value \(x\). The subscript indicates which feature value indexes the response law. In a fixed-design experiment, \(x_i\) is a constant, not a random variable on which one is conditioning. Thus \(Y_i\mid x_i\sim F_{x_i}\) is sometimes used as shorthand for “the response at the prescribed point \(x_i\) has distribution \(F_{x_i}\).”
If the response has a finite first moment under \(F_x\), define its feature-indexed mean by
Consequently, \(E[Y_i]=\mu(x_i)\). If second moments are finite, its variance at that point is \(v(x_i)=\operatorname{Var}(Y_i)\). The points \(x_i\) may repeat. Under a common response mechanism, responses at repeated feature values have the same marginal distribution and mean; this statement alone says nothing about whether those responses are independent.
The fixed design also has a joint response law: a probability distribution for the whole vector \((Y_1,\ldots,Y_n)\). Specifying each marginal distribution \(F_{x_i}\) does not in general specify that joint law. Dependence among responses is a separate part of the experiment.
Worked Example: Normal Responses at Four Prescribed Points
Suppose the prescribed points are \(x_1=0,\ x_2=1,\ x_3=1,\ x_4=2\). At each point \(x\), let the response have distribution \(F_x=N(1+2x,1+x)\), where the second argument is the variance. Find each marginal mean and variance, the expected average response, and the expected sum of squared deviations from the point-specific means.
The means are \(\mu(x)=1+2x\), so \(\mu(x_1)=1,\ \mu(x_2)=3,\ \mu(x_3)=3,\ \mu(x_4)=5\). The variances are \(v(x)=1+x\), giving \(1,2,2,3\), respectively. In particular, the repeated point \(x=1\) has the same marginal distribution \(N(3,2)\) on both visits.
By linearity of expectation, \(E[(Y_1+Y_2+Y_3+Y_4)/4]=(1+3+3+5)/4=3\). For each \(i\), \(E[(Y_i-\mu(x_i))^2]=\operatorname{Var}(Y_i)=v(x_i)\). Therefore the expected sum of squared deviations is \(1+2+2+3=8\). These calculations require the stated marginal means and variances, not independence.
Worked Example: Binary Responses at Fixed Points
A study prescribes \(x_1=0,\ x_2=0,\ x_3=1\). At point \(x=0\), responses are Bernoulli with success probability \(0.2\); at \(x=1\), they are Bernoulli with success probability \(0.6\). Find the expected number of successes and the expected success proportion.
For a Bernoulli response with success probability \(p\), its mean is \(p\). Thus the three response means are \(0.2,0.2,0.6\), and the expected number of successes is \(E[Y_1+Y_2+Y_3]=0.2+0.2+0.6=1\). The expected proportion is \(1/3\), since the sample size is three and the proportion is \((Y_1+Y_2+Y_3)/3\).
The corresponding variances are \(p(1-p)\): \(0.2(0.8)=0.16\), \(0.16\), and \(0.6(0.4)=0.24\). The two observations at \(x=0\) share a marginal law under this setup. Their joint behavior is not determined by their Bernoulli marginals.
Means and Covariances of the Response Vector
The vector notation makes clear what the fixed-design specification says about the responses. Let \(\boldsymbol{\mu}\) be the vector of point-specific means, and let \(\Sigma\) be the covariance matrix of the response vector. Suppose each response has a finite second moment.
To verify the result, take expectations component by component. Since \(E[Y_i]=\mu(x_i)\), it follows that \(E[\mathbf{Y}]=\boldsymbol{\mu}\). By definition, the \((i,j)\) entry of the covariance matrix is \(E[(Y_i-E[Y_i])(Y_j-E[Y_j])]\), which is \(\operatorname{Cov}(Y_i,Y_j)\). Setting \(j=i\) gives \(\operatorname{Var}(Y_i)=v(x_i)\). The marginal distribution of each response fixes its diagonal variance, but does not by itself fix the covariance between two different responses. \(\square\)
Worked Example: The Same Marginals With Dependent Responses
Consider two prescribed points \(x_1=0\) and \(x_2=1\), with mean function \(\mu(x)=2+3x\). Let a single random variable \(\varepsilon\sim N(0,4)\) be shared by both responses: \(Y_1=2+\varepsilon\) and \(Y_2=5+\varepsilon\). Find their marginal distributions, covariance, and the variance of their difference.
Adding constants shifts the mean but does not change the variance. Hence \(Y_1\sim N(2,4)\) and \(Y_2\sim N(5,4)\), as specified by the point-specific means and variances. Their covariance is \(\operatorname{Cov}(Y_1,Y_2)=\operatorname{Cov}(2+\varepsilon,5+\varepsilon)=\operatorname{Var}(\varepsilon)=4\).
Their difference is \(Y_2-Y_1=(5+\varepsilon)-(2+\varepsilon)=3\), so \(\operatorname{Var}(Y_2-Y_1)=0\). Equivalently, \(\operatorname{Var}(Y_2-Y_1)=4+4-2(4)=0\). The responses have the stated marginals but are not independent: knowing one determines the other. Prescribing the feature points does not rule out this dependence.
Expected Squared Loss at the Prescribed Points
A predictor \(g\) assigns a fixed prediction \(g(x_i)\) to each prescribed point. When the response has finite second moments, the expected average squared loss has a direct decomposition into response variability and squared discrepancy between the predictor and the point-specific means.
For each observation, add and subtract its mean: \(Y_i-g(x_i)=(Y_i-\mu(x_i))+(\mu(x_i)-g(x_i))\). Squaring and taking expectations gives
because \(E[Y_i-\mu(x_i)]=0\). Taking expectations of the finite sum and dividing by \(n\) proves the result. This calculation does not require independent responses; the loss is a sum of separate squared terms, so its expectation is the sum of their expectations. \(\square\)
Worked Example: Comparing a Fixed Predictor With the Point Means
Use the four-point design from the first example, with means \(1,3,3,5\) and variances \(1,2,2,3\). Suppose a predictor returns \(g(x)=2+2x\), so its predictions at the four points are \(2,4,4,6\). Calculate its expected average squared loss.
The squared mean discrepancies are \((1-2)^2=1,\ (3-4)^2=1,\ (3-4)^2=1,\ (5-6)^2=1\). The sum of response variances is \(1+2+2+3=8\), and the sum of squared mean discrepancies is \(4\). Therefore the theorem gives expected average squared loss \((8+4)/4=3\).
The value 3 includes both irreducible response variation at these points, whose average is \(8/4=2\), and the predictor's squared discrepancy from the means, whose average is \(4/4=1\). The calculation is tied to these prescribed points; it does not require describing a distribution over feature values.
Common Pitfall: Treating Fixed Design as an Independence Assumption
A frequent error is to reason: “The feature values are fixed, so the response observations must be independent,” or to assume that all observations are identically distributed. Neither conclusion follows. Fixed design says that \(x_1,\ldots,x_n\) are prescribed constants. Responses at different points can have different marginal distributions, and responses—even at repeated points—can be dependent.
The correction is to keep three parts of the specification distinct: the list of prescribed points, the marginal response law at each point, and the joint dependence structure of the response vector. Independence or identical distribution may be added as assumptions when justified, but they are not consequences of a fixed design.
Check Your Understanding
Use the definitions and results above to answer these questions.
- In a fixed-design experiment, which quantities are constants and which are random?
- If \(x_2=x_5\) and both responses follow a common response mechanism, what can be concluded about their marginal distributions? What cannot be concluded from that fact alone?
- Suppose \(E[Y_i]=\mu(x_i)\) and \(\operatorname{Var}(Y_i)=v(x_i)\). What are the diagonal entries of the response covariance matrix?
- Why does the expected average squared loss formula not require independence?
- Give one additional assumption that would specify something about the joint response behavior beyond the marginal laws.