Extrapolation Beyond Observed Covariate Support

Why Extrapolation Raises an Identification Problem

A prediction rule can return a number at any input in its domain. That computational ability does not mean the observed data determine the number. If observations come from a limited range of covariates, values outside that range may never be tested by the experiment. Different response mechanisms can then agree on everything the experiment observes and still give different predictions elsewhere.

The central issue is not whether a particular extrapolation looks plausible. It is whether the data-generating law is sufficient to distinguish it from alternatives. We will make that question precise by comparing laws that induce exactly the same distribution for the observed data.

Definition: For a random-design experiment, the population support of \(X\), denoted \(\operatorname{supp}(P_X)\), is the set of covariate values \(x\) such that every open neighborhood of \(x\) has positive probability under the distribution \(P_X\). In particular, a point outside this support has some open neighborhood that the experiment visits with probability zero. The observed covariate values are the values actually recorded in a particular sample; they need not equal the population support.

The distinction matters. A finite sample from a continuous distribution will usually contain no repeated covariate values, even when every point in an interval belongs to the population support. A prediction at a point between observed values may therefore be interpolation relative to the population support, not extrapolation beyond it. Conversely, a point outside the population support cannot be reached by collecting more observations from the same distribution.

For fixed-design regression, the experiment instead uses specified covariate values. The relevant observed region is then the set of design points, unless the target is explicitly defined over some larger region. In either setting, the identification question is whether different response laws could have produced the same observed-data distribution.

Indistinguishable Laws and Unidentified Targets

Suppose the response distribution given a covariate is described by a conditional distribution \(P(Y\mid X=x)\). The observed joint law only uses these conditional distributions at covariate values that occur with positive probability. More precisely, it determines the conditional law only up to changes on sets of \(P_X\)-probability zero. The following result states the consequence directly.

Proposition: Let \(X\) and \(Y\) take values in Euclidean spaces, with common marginal distribution \(P_X\) under two candidate laws \(P\) and \(P'\). Suppose their conditional response distributions \(P(Y\mid X=x)\) and \(P'(Y\mid X=x)\) agree for \(P_X\)-almost every \(x\). Then \(P\) and \(P'\) induce the same joint distribution of \((X,Y)\). Consequently, an independent sample of size \(n\) has the same distribution under both laws for every \(n\).

Proof. Write \(K(x,B)=P(Y\in B\mid X=x)\) and \(K'(x,B)=P'(Y\in B\mid X=x)\) for the conditional probability distributions, and let \(A\) be a measurable set of covariate values and \(B\) a measurable set of response values. By the definition of a conditional distribution,

P(X\in A,Y\in B)=\int_A K(x,B)\,dP_X(x).

The same formula holds under \(P'\), with \(K'\) in place of \(K\). Since \(K(x,B)=K'(x,B)\) for \(P_X\)-almost every \(x\), the two integrals are equal. Thus the laws agree on all measurable rectangles \(A\times B\), which determine the joint distribution on the product of Euclidean spaces. For an independent sample, the sample law is the \(n\)-fold product of the joint law, so it too is identical under \(P\) and \(P'\). \(\square\)

This proposition gives a precise test for identification. A target is identified by an experiment only if it has the same value under any two laws that induce the same observed-data distribution. If two such laws give different conditional means at an extrapolation point, then the mean at that point is not identified from the experiment alone.

Key distinction: A model or other assumption may select one extrapolation from several possibilities. That does not mean the observed data alone identified it. Identification concerns what is shared by all laws compatible with the observed-data distribution.

Worked Examples of Extrapolation

Worked Example: A Covariate With Two Possible Values

Suppose \(X\) takes the values 0 and 1, each with probability \(1/2\). Consider two candidate response laws. Under both, \(Y\mid X=x\) is normal with variance 1. Under law \(P\), its mean is 0 for every \(x\). Under law \(P'\), its mean is 0 at \(x=0\) and \(x=1\), but 5 at \(x=2\).

The population support is \(\{0,1\}\), so the conditional distributions agree at every covariate value that can occur. For \(x=0\) or \(x=1\), both laws give \(Y\mid X=x\sim N(0,1)\). Thus they induce exactly the same observed-data distribution, for one observation or any sample size.

At \(x=2\), however, the conditional means differ: they are 0 under \(P\) and 5 under \(P'\). No sample from this experiment can contain \(X=2\), because \(P(X=2)=0\). Therefore the observed data do not determine the mean at \(x=2\). The value 5 is not contradicted by the data, but neither is it supported over the alternative value 0.

Worked Example: A Target Region Outside a Continuous Support

Let \(X\sim\operatorname{Uniform}(0,1)\), and suppose \(Y\mid X=x\) is normal with variance 1. Compare two conditional mean functions:

m_0(x)=x,\qquad m_1(x)=x+(x-1)_+,

where \((u)_+=\max(u,0)\). On the population support \([0,1]\), \((x-1)_+=0\), so \(m_0(x)=m_1(x)\) throughout the support. The two laws therefore give the same distribution of every observed sample from \(X\sim\operatorname{Uniform}(0,1)\).

Now consider a target covariate distribution \(Q=\operatorname{Uniform}(1.5,2)\). The average conditional mean under \(m_0\) is

\int_{1.5}^{2}x\,\frac{1}{0.5}\,dx =2\left[\frac{x^2}{2}\right]_{1.5}^{2} =4-2.25 =1.75.

Under \(m_1\), the added term has target average

\int_{1.5}^{2}(x-1)\,\frac{1}{0.5}\,dx =2\left[\frac{(x-1)^2}{2}\right]_{1.5}^{2} =1-0.25 =0.75.

Hence the target average under \(m_1\) is \(1.75+0.75=2.50\), not 1.75. The observed training law is identical in the two cases, while the target averages differ. A target distribution can therefore make an extrapolation problem especially clear: it may ask for a summary in a region where the training experiment provides no observations.

Worked Example: Two Fixed-Design Laws Agree at Every Design Point

Take a fixed design with \(x_1=0\) and \(x_2=1\). Suppose the responses are independent normal variables with variance 1. Under the first law, the conditional mean function is \(m_0(x)=0\). Under the second, let \(m_1(x)=x(x-1)\).

At the design points, \(m_1(0)=0(0-1)=0\) and \(m_1(1)=1(1-1)=0\). Thus under either law the observed responses are independent \(N(0,1)\) variables, and the full fixed-design data distribution is the same.

At the target point \(x=2\), the first mean is \(m_0(2)=0\), whereas the second is \(m_1(2)=2(2-1)=2\). The design data cannot distinguish these values. The calculation exhibits the same identification issue in a fixed-design experiment: matching the response laws at every design point does not force the functions to match elsewhere.

Why Indistinguishability Cannot Be Fixed by a Larger Sample

When two laws produce identical data distributions, the problem is not merely that a small sample may be noisy. No increase in sample size can distinguish the laws using that experiment: the distribution of the entire sample remains the same. In particular, a data-based estimator cannot consistently estimate a target that takes different fixed values under the two laws.

Result: Suppose two laws give the same distribution for the observed data at every sample size, but a target equals \(a\) under the first law and \(b\ne a\) under the second. No estimator based only on those data can converge in probability to the correct target under both laws.

Proof. Let \(T_n\) be any estimator computed from the data. Because the data distributions are identical, \(T_n\) has the same distribution under both laws. Choose \(\varepsilon=|a-b|/3\), so the events \(\{|T_n-a|<\varepsilon\}\) and \(\{|T_n-b|<\varepsilon\}\) are disjoint. If \(T_n\) converged in probability to \(a\) under the first law, the probability of the first event would tend to 1. If it also converged in probability to \(b\) under the second law, the probability of the second event, under that law, would tend to 1. Since the distribution of \(T_n\) is the same under both laws, these two disjoint events would both have probability tending to 1 under one common distribution. Their probabilities cannot sum to more than 1, a contradiction. \(\square\)

The result does not say that every extrapolation is useless or that a prediction rule cannot output a value outside the observed region. It says that the value is not determined by these data without additional restrictions on the possible laws. For example, a chosen functional form or a smoothness condition may imply a particular continuation. Such restrictions can be scientifically or practically useful, but they add information beyond the observed-data distribution.

Common Pitfall: Confusing Plausibility With Identification

A common mistake is to argue: “The fitted relationship is smooth, so its continuation beyond the observed range is determined.” Smoothness of the fitted curve is a property of the chosen fit, not a consequence of the observed data unless smoothness has been imposed as a restriction on the admissible laws. Even continuity alone need not determine the continuation: the two continuous mean functions \(m_0(x)=x\) and \(m_1(x)=x+(x-1)_+\) agree on \([0,1]\) but differ for \(x>1\).

Another mistake is to treat the largest and smallest values in a finite sample as the limits of the population support. A sample might observe only values between 0.2 and 0.8 even when the population support is \([0,1]\). Lack of observed values near a point can make prediction difficult, but it is not the same as a population region that the experiment can never reach. Conversely, even infinitely many observations cannot reveal a response law on a region with zero covariate probability.

The practical question is therefore two-part: first, where does the data-generating covariate distribution place probability; second, what additional assumptions, if any, justify predictions in regions it does not cover? Keeping those questions separate prevents a model’s extrapolated answer from being mistaken for a data-identified fact.

Check Your Understanding

Use the distinction between observed data and extrapolated targets to answer these questions.

In a random-design experiment, why can two conditional response laws differ outside \(\operatorname{supp}(P_X)\) yet induce the same observed-data law?
Can a point between the smallest and largest observed covariates still be outside the population support? Can a point inside that observed range still be outside the population support?
For the two mean functions in the continuous-support example, what are their values at \(x=1.5\), and why can the data not distinguish them?
In the fixed-design example, verify the two mean functions agree at \(x=0\) and \(x=1\), but not at \(x=2\).
What extra role does a smoothness or model restriction play when predicting beyond the covariate support?