Weighted Series Least Squares for Nonuniform Designs

Why Weight a Series Fit?

In random-design regression, observations are more common where the covariate density is high. Ordinary least squares therefore gives those regions more influence on its fitted series. That is appropriate when the goal is to minimize prediction error averaged over the sampling distribution. It is not generally appropriate when the goal is an integrated squared error under a different reference measure, such as uniform measure over a region. Inverse-density weighting changes the population criterion so that the intended target—not the accidental concentration of the sample—determines the projection.

We derive the weighting rule for a finite series and show both what it targets and when the empirical coefficients approach that target. The key distinction is between the distribution that generates the covariates and the measure under which approximation error is evaluated.

Setup and Definition

Let $X$ take values in a measurable domain $D\subseteq\mathbb{R}^d$, with known sampling density $p$ relative to Lebesgue measure. Given $X=x$, let the response $Y$ have conditional mean $m(x)=E[Y\mid X=x]$. Fix $K$ basis functions and write $\phi(x)=(\phi_1(x),\ldots,\phi_K(x))^\top$. A series function has the form $g_\beta(x)=\phi(x)^\top\beta$, where $\beta\in\mathbb{R}^K$.

Let $q$ be a target density on $D$. The corresponding integrated squared-error approximation criterion is $\int_D (m(x)-g_\beta(x))^2q(x)\,dx$. For a uniform target on a domain of finite positive volume, $q(x)=1/|D|$. Multiplying this target density by a positive constant would not change the minimizing coefficients.

Definition: The inverse-density weighted series least-squares estimator minimizes $$ \widehat Q_n(\beta)=\frac{1}{n}\sum_{i=1}^n \frac{q(X_i)}{p(X_i)}\bigl(Y_i-\phi(X_i)^\top\beta\bigr)^2. $$ The factor $q(X_i)/p(X_i)$ converts averages under the sampling density $p$ into averages under the target density $q$. For a uniform target on $D$, the weight is $1/(|D|p(X_i))$; the constant $1/|D|$ can be omitted without changing the minimizer.

The ratio is defined where $p(x)>0$. A necessary support condition is $q(x)=0$ wherever $p(x)=0$, up to sets of Lebesgue measure zero: the sample cannot represent a target region it never visits. Other integrability and nonsingularity conditions are needed for the formulas below and will be stated with the results.

Population Criterion and Normal Equations

Write $w(x)=q(x)/p(x)$ where $p(x)>0$. If $E[Y^2\mid X=x]$ is integrable under $q$, then conditioning on $X$ gives the population weighted criterion

Q(\beta)=E\!\left[w(X)\{Y-\phi(X)^\top\beta\}^2\right] =\int_D q(x)E\!\left[\{Y-g_\beta(x)\}^2\mid X=x\right]dx.

The identity follows because the density of $X$ in the expectation is $p$, so $w(x)p(x)=q(x)$. If $v(x)=\operatorname{Var}(Y\mid X=x)$, the conditional squared-error decomposition yields

Q(\beta)=\int_D q(x)\left[\{m(x)-g_\beta(x)\}^2+v(x)\right]dx.

The variance term does not depend on $\beta$. Thus minimizing $Q$ gives the projection of $m$ onto the chosen series span under the target density $q$, not under the sampling density $p$.

Theorem: Suppose $p>0$ wherever $q>0$, $\int_D q(x)E[Y^2\mid X=x]\,dx<\infty$, and the matrix $$ Q_q=\int_D \phi(x)\phi(x)^\top q(x)\,dx $$ is finite and positive definite. Then $Q(\beta)$ has the unique minimizer $$ \beta_q=Q_q^{-1}b_q,\qquad b_q=\int_D\phi(x)m(x)q(x)\,dx. $$

Proof. The conditional decomposition above shows that minimizing $Q(\beta)$ is equivalent to minimizing $\int_D (m(x)-\phi(x)^\top\beta)^2q(x)\,dx$. Expanding this expression gives

\int_D m(x)^2q(x)\,dx-2\beta^\top b_q+\beta^\top Q_q\beta.

Its gradient with respect to $\beta$ is $-2b_q+2Q_q\beta$. Setting the gradient equal to zero gives $Q_q\beta=b_q$, whose unique solution is $Q_q^{-1}b_q$ because $Q_q$ is positive definite. Positive definiteness also makes the quadratic strictly convex, so this solution is the unique minimizer. $\square$

The same result follows by differentiating the weighted population residual criterion. Its normal equations are $E[w(X)\phi(X)\{Y-\phi(X)^\top\beta\}]=0$. The two key expectations are

E[w(X)\phi(X)\phi(X)^\top]=Q_q,\qquad E[w(X)\phi(X)Y]=b_q.

For the second equality, use $E[Y\mid X=x]=m(x)$. These equations make explicit why the density ratio, rather than an arbitrary reweighting, recovers the target projection.

Worked Calculations

Worked Example: Uniform Projection Under a Nonuniform Design

Let $D=[0,1]$, let the sampling density be $p(x)=\frac{2}{3}(1+x)$, and take a uniform target $q(x)=1$. Indeed, $\int_0^1 p(x)\,dx=\frac{2}{3}(1+\frac12)=1$. Suppose $Y=X^2$ and use the basis $\phi(x)=(1,x)^\top$. The target is the best linear approximation to $x^2$ under uniform measure.

The target Gram matrix and right-hand side are

Q_q= \begin{pmatrix} \int_0^1 1\,dx & \int_0^1 x\,dx\\ \int_0^1 x\,dx & \int_0^1 x^2\,dx \end{pmatrix} = \begin{pmatrix}1&1/2\\1/2&1/3\end{pmatrix}, \qquad b_q= \begin{pmatrix}\int_0^1 x^2\,dx\\\int_0^1 x^3\,dx\end{pmatrix} = \begin{pmatrix}1/3\\1/4\end{pmatrix}.

The normal equations are $a+b/2=1/3$ and $a/2+b/3=1/4$, where $a$ is the intercept and $b$ is the slope. Substituting $a=1/3-b/2$ into the second equation gives $1/6+b/12=1/4$, hence $b=1$ and $a=-1/6$. The weighted population fit is therefore $g(x)=-1/6+x$. The sampling density affects which points are observed more often, but the weights recover the uniform-target projection.

Worked Example: What Ordinary Least Squares Targets Instead

Keep $D=[0,1]$, $p(x)=\frac{2}{3}(1+x)$, $Y=X^2$, and the basis $(1,x)^\top$, but now use ordinary least squares. Its population criterion averages under $p$, so the relevant moments are

E_p[X]=\frac{5}{9},\qquad E_p[X^2]=\frac{7}{18},\qquad E_p[X^3]=\frac{3}{10}.

For example, $E_p[X^k]=\frac{2}{3}\{\frac{1}{k+1}+\frac{1}{k+2}\}$, which gives each displayed value. The slope of the best linear fit under $p$ is the covariance of $X$ and $X^2$ divided by the variance of $X$:

\operatorname{Var}_p(X)=\frac{7}{18}-\left(\frac{5}{9}\right)^2=\frac{13}{162}, \qquad \operatorname{Cov}_p(X,X^2)=\frac{3}{10}-\frac{5}{9}\frac{7}{18}=\frac{34}{405}.

Thus the slope is $(34/405)/(13/162)=68/65$. The intercept is $E_p[X^2]-(68/65)E_p[X]=7/18-68/117=-5/26$. Ordinary least squares gives $-5/26+(68/65)x$, not the uniform-target fit $-1/6+x$. Both are valid projections, but they answer different approximation questions.

Worked Example: Choosing a Nonuniform Target

Suppose the same sampling density $p(x)=\frac{2}{3}(1+x)$ is used, but the target density is $q(x)=2x$ on $[0,1]$. This is a probability density because $\int_0^1 2x\,dx=1$. For $Y=X^2$ and the linear basis, its moments are $E_q[X]=2/3$, $E_q[X^2]=1/2$, and $E_q[X^3]=2/5$.

The target variance and covariance are $1/2-(2/3)^2=1/18$ and $2/5-(2/3)(1/2)=1/15$. Therefore the target projection has slope $(1/15)/(1/18)=6/5$ and intercept $1/2-(6/5)(2/3)=-3/10$. The appropriate weight is

\frac{q(x)}{p(x)} =\frac{2x}{(2/3)(1+x)} =\frac{3x}{1+x}.

Indeed, multiplying this weight by the sampling density gives $p(x)\{q(x)/p(x)\}=q(x)$. The weighted fit therefore targets $-3/10+(6/5)x$, while the uniform-target fit from the first example is different. The target measure must be chosen to match the question of interest.

Empirical Coefficients and Convergence

For data $(X_i,Y_i)_{i=1}^n$, define the weighted empirical Gram matrix and right-hand side by

\widehat Q_n=\frac{1}{n}\sum_{i=1}^n w(X_i)\phi(X_i)\phi(X_i)^\top,\qquad \widehat b_n=\frac{1}{n}\sum_{i=1}^n w(X_i)\phi(X_i)Y_i.

Whenever $\widehat Q_n$ is invertible, differentiating the empirical criterion gives $\widehat\beta_n=\widehat Q_n^{-1}\widehat b_n$. This is the sample version of the target normal equations; it does not assert that the finite-sample fitted function equals the population projection.

Theorem: Suppose $(X_i,Y_i)$ are iid, $p>0$ wherever $q>0$, and $Q_q$ is finite and positive definite. Assume every entry of $w(X)\phi(X)\phi(X)^\top$ and $w(X)\phi(X)Y$ has finite absolute expectation. Then $\widehat Q_n\to Q_q$ and $\widehat b_n\to b_q$ almost surely. Consequently, $\widehat Q_n$ is eventually invertible almost surely and $\widehat\beta_n\to\beta_q$ almost surely.

Proof. By the density-ratio identities, the expectations of the summands in $\widehat Q_n$ and $\widehat b_n$ are $Q_q$ and $b_q$. The assumed absolute integrability permits the strong law of large numbers to be applied to each of the finitely many matrix and vector entries. Hence the stated almost-sure convergences hold entrywise, and therefore also in finite-dimensional matrix and vector norms.

Since $Q_q$ is positive definite, its smallest eigenvalue is positive. Norm convergence implies that, eventually, the smallest eigenvalue of $\widehat Q_n$ is at least half that of $Q_q$, so $\widehat Q_n$ is eventually invertible. Matrix inversion is continuous at an invertible matrix; thus

\widehat\beta_n=\widehat Q_n^{-1}\widehat b_n \longrightarrow Q_q^{-1}b_q=\beta_q \quad\text{almost surely}.

This establishes convergence of the finite-dimensional coefficient vector under the stated conditions. It does not remove the need for a suitable basis or guarantee small approximation error for a particular $K$. $\square$

Practical Interpretation and a Common Pitfall

Inverse-density weighting gives less total influence to regions that are densely sampled and more to regions that are sparsely sampled, as required to represent the target measure. For a uniform target, the weight is proportional to $1/p(X_i)$. If the target itself is nonuniform, the correct ratio is $q(X_i)/p(X_i)$, not simply $1/p(X_i)$. In either case, the population normal equations explain exactly which projection is being estimated.

Common mistake: “The observations are drawn from $p$, so ordinary least squares must estimate the uniform integrated-error projection.” This confuses the sampling distribution with the target measure. Ordinary least squares projects under $p$; it projects under $q$ only when the target and sampling densities agree, or in special cases where the projections happen to coincide.
Correction: Write the desired target integral first. The ratio $q/p$ makes its expectation under the sampling distribution equal that integral. Check that the sampling density is positive wherever the target assigns mass and that the weighted moments are integrable.

Weights can become large where $p$ is small. The identities for the population target do not by themselves guarantee stable finite-sample estimates: the empirical conditions above, including integrability, still matter. Weighting corrects the target of the fit; it does not create observations in poorly sampled regions.

Check Your Understanding

Use the target density, not just the data-generating density, to answer each question.

What weighted population criterion corresponds to integrated squared error under a target density $q$ when $X$ is sampled with density $p$?
Why does the conditional variance of $Y$ not change the population minimizing coefficient vector?
For a uniform target on a domain of volume $|D|$, what is the density-ratio weight, and why can its constant factor be omitted?
What fails if $q(x)>0$ on a region where $p(x)=0$?
Under what assumptions does the empirical weighted coefficient vector converge to the target projection?