Covariate Distributions and the Regions a Target Covers

Why the Feature Distribution Matters

A population target is tied to a population distribution. In random-design sampling, each feature vector $X$ is drawn from a distribution $P_X$, and covariate regions occur with probabilities determined by that distribution. A region that has large probability contributes frequently to population averages; a region with probability zero contributes nothing to such averages. This is different from treating every possible covariate value as equally represented.

The support of $P_X$ describes the covariate locations around which the population places probability. It is useful to distinguish support from finite-sample coverage: even a region with positive population probability may not appear in a particular sample. The support tells us which regions are represented in the population, while their probabilities help determine how often they appear in a sample.

Definition: Let $X$ take values in $\mathbb{R}^d$, with distribution $P_X$. The support of $P_X$, written $\operatorname{supp}(P_X)$, is the set of points $x$ such that every open ball around $x$ has positive probability: $$ \operatorname{supp}(P_X) = \{x\in\mathbb{R}^d: P_X(B(x,r))>0\text{ for every }r>0\}. $$ Here $B(x,r)=\{z:\|z-x\|<r\}$. Thus, a point belongs to the support if the population assigns positive probability to every neighborhood around it; the point itself need not have positive probability.

Support, Point Probability, and Region Weight

For a discrete feature, a support point typically has positive probability itself. For a continuous feature, an individual point can have probability zero while still belonging to the support, because every neighborhood of that point has positive probability. This distinction is essential: support concerns neighborhoods, not just individual-point probabilities.

A population target that averages a quantity $q(X)$ is weighted by the feature distribution: $$ E[q(X)] = \int q(x)\,dP_X(x). $$ For any measurable region $A$, the probability $P_X(A)$ determines how much population mass lies there. If $P_X(A)=0$, the region contributes no weight to this expectation. If $P_X(A)>0$, it can contribute, with its influence depending both on its probability and on the values of $q$ there.

This weighting is one way to see the distinction between population targets and fixed-design targets. A fixed-design target may assign weight to a specified finite set of covariate values. A population target instead weights covariate values according to $P_X$. In particular, the population target does not generally weight all covariate values equally.

Worked Example: A Discrete Feature Distribution

Suppose $X$ takes the values $A,B,C$ with probabilities $0.5,0.3,0.2$, respectively. Let $q(A)=2$, $q(B)=5$, and $q(C)=10$. Find the population average $E[q(X)]$, identify the support, and calculate the probability that a sample of five independent observations contains no $C$.

Population average. Each value is weighted by its probability: $$ E[q(X)] =0.5(2)+0.3(5)+0.2(10) =1+1.5+2 =4.5. $$

Support. Each of $A,B,C$ has positive probability, so each is in the support. There are no other possible feature values in this distribution, and the support consists of these three points.

Sample coverage. One observation has probability $0.8=0.5+0.3$ of not being $C$. By independence, all five observations avoid $C$ with probability $$ 0.8^5=0.32768. $$ Thus the probability that the sample contains at least one $C$ is $1-0.32768=0.67232$. Although $C$ is in the population support, it is not guaranteed to occur in a sample of five.

Continuous Distributions and Boundary Points

For a continuous feature, probabilities are assigned to regions rather than to individual points. If $X$ is uniform on $[0,1]$, then $P(X=x)=0$ for every individual $x$, including $x=0.4$. Nevertheless, every open neighborhood of $0.4$ intersects $[0,1]$ in an interval of positive length, so $0.4$ belongs to the support.

The endpoints $0$ and $1$ also belong to the support. For example, for any $r>0$, the intersection of $B(0,r)$ with $[0,1]$ contains an interval of positive length. Yet $P(X=0)=0$. This illustrates why the definition uses neighborhoods.

Worked Example: A Continuous Feature Distribution

Let $X\sim \operatorname{Uniform}[0,1]$, and take $q(x)=x^2$. Find the support and the population average $E[q(X)]$. Then calculate the probability that a sample of 20 independent observations contains no feature value in $[0,0.1]$.

Support. Every neighborhood of every point in $[0,1]$, including its endpoints, has positive probability. A point outside $[0,1]$ has a sufficiently small neighborhood that does not intersect $[0,1]$. Therefore, $\operatorname{supp}(P_X)=[0,1]$.

Population average. The uniform density is $1$ on $[0,1]$, so $$ E[X^2]=\int_0^1 x^2\,dx =\left[\frac{x^3}{3}\right]_0^1 =\frac{1}{3}. $$ The average depends on the distribution across the interval, not on a sum that gives each point an equal positive weight.

Sample coverage. The interval $[0,0.1]$ has probability $0.1$. Each observation misses it with probability $0.9$, so 20 independent observations all miss it with probability $$ (1-0.1)^{20}=0.9^{20}\approx 0.1216. $$ The probability of at least one observation in the interval is therefore approximately $1-0.1216=0.8784$. The interval has positive population mass, but finite-sample coverage is still not certain.

Two Consequences of the Support Definition

The definition gives two useful facts: regions outside the support have zero probability, and neighborhoods of support points have positive probability. The first fact requires a small technical observation because there can be uncountably many points outside the support.

Result: If $X$ takes values in $\mathbb{R}^d$, then $P_X(\mathbb{R}^d\setminus\operatorname{supp}(P_X))=0$. In particular, the feature distribution places probability one on its support.

Proof. If $x$ is outside the support, the definition implies that some open ball around $x$ has probability zero. The open balls with rational centers and rational radii form a countable base for the open sets of $\mathbb{R}^d$. Therefore, $x$ lies in a rational ball of probability zero contained in a zero-probability neighborhood of $x$. The complement of the support is consequently covered by the collection of all rational balls that have probability zero. This collection is countable, and a countable union of probability-zero sets has probability zero. Hence the complement of the support has probability zero. $\square$

This result does not mean that every point in the support must have positive point probability. The uniform example shows otherwise. It means that the distribution assigns probability one to the support as a whole.

Result: Let $X_1,\ldots,X_n$ be independent observations with the same feature distribution $P_X$, and let $A$ be a measurable region with $p=P_X(A)$. The probability that at least one sampled feature lies in $A$ is $1-(1-p)^n$.

Proof. Each observation falls outside $A$ with probability $1-p$. Independence gives $$ P(X_1\notin A,\ldots,X_n\notin A) =\prod_{i=1}^n P(X_i\notin A) =(1-p)^n. $$ The event that at least one observation falls in $A$ is the complement of the event that all observations miss $A$. Its probability is therefore $1-(1-p)^n$. If $p=0$, this probability is zero for every $n$; if $p>0$, it approaches one as $n$ increases. $\square$

For a support point $x$, every open ball $B(x,r)$ has positive probability by definition. Applying the result to that ball shows that the chance of seeing at least one observation near $x$ increases with sample size. It does not imply that a sample contains the exact point $x$, which may have probability zero.

Worked Example: A Mixed Feature Distribution

Suppose $X$ has probability $0.4$ at $0$, and with probability $0.6$ is drawn uniformly from $[1,3]$. Let $q(0)=4$, and let $q(x)=x$ for $x\in[1,3]$. Find the support and $E[q(X)]$. Also find the probability that ten independent observations include at least one feature in $[1,1.5]$.

Support. The isolated point $0$ is in the support because every neighborhood of it has probability at least $0.4$. Every point in $[1,3]$ is also in the support: each neighborhood intersects the uniform interval in a set of positive length. Points strictly between $0$ and $1$, and points outside $\{0\}\cup[1,3]$, have a neighborhood with probability zero. Thus the support is $\{0\}\cup[1,3]$.

Population average. Conditional on drawing from the uniform component, $E[X]= (1+3)/2=2$. Hence $$ E[q(X)] =0.4(4)+0.6(2) =1.6+1.2 =2.8. $$ The isolated atom and the continuous interval both contribute, each weighted by its probability.

Sample coverage. The interval $[1,1.5]$ has length $0.5$ within a uniform interval of length $2$, so its conditional probability under the uniform component is $0.5/2=0.25$. Its unconditional probability is $0.6(0.25)=0.15$. The probability that ten observations all miss it is $0.85^{10}\approx0.1969$, so the probability that at least one falls in it is $$ 1-0.85^{10}\approx 0.8031. $$ The gap between $0$ and $1$ has probability zero; the interval $[1,1.5]$, by contrast, has positive probability despite containing individual points of probability zero.

What a Population Target Does—and Does Not—Cover

For an integrated population target, the feature distribution determines the regions that receive weight. Changing a quantity $q(x)$ only on a set $A$ with $P_X(A)=0$ does not change its expectation, provided the quantities are integrable. In particular, values assigned outside the support cannot affect $E[q(X)]$, since the complement of the support has probability zero.

The converse requires care: support membership does not give equal weight to every location. In a continuous distribution, individual points have zero probability, and a narrow region may have far less weight than a broad one. In a discrete distribution, an atom’s weight is its point probability. Thus, “in the support” means represented through positive-probability neighborhoods, not “equally likely” or “certain to appear in the observed sample.”

A common mistake is to assume that a point in the support must occur in every sample—or even that a continuous sample must contain any particular support point. The correction is to ask about the probability of a region: for a region $A$, the chance of seeing it in $n$ independent observations is $1-(1-P_X(A))^n$. A point can be in the support while having probability zero, and a positive-probability region can still be missed by a finite sample.

Support is therefore a population-level description of where covariate probability is concentrated, while finite-sample coverage is a probability calculation. Keeping those ideas separate makes it possible to interpret population targets without mistaking a possible covariate value for a frequently observed one.

Check Your Understanding

Answer each question using the distinction between support, point probability, and region probability.

If $X$ is uniform on $[0,1]$, why is $0.5$ in the support even though $P(X=0.5)=0$?
A discrete feature takes values $a$ and $b$ with probabilities $0.7$ and $0.3$. What is the probability that a sample of four independent observations contains no $b$?
For an integrable $q$, why does changing $q$ outside the support leave $E[q(X)]$ unchanged?
A region $A$ has probability $0.02$. Write the probability that at least one of $n$ independent observations falls in $A$.
In the mixed-distribution example, explain why the gap between $0$ and $1$ is not in the support, while $1$ is.