Regression Series Estimators and Basis Coefficients

Finite-Dimensional Regression by Basis Expansion

A regression function can be approximated using a finite collection of known functions, even when it is not assumed to belong to a simple parametric family. The central calculation is least squares: choose the coefficients that make the resulting linear combination of basis functions fit the observed responses as closely as possible. It is important to distinguish this sample calculation from the population approximation defined using the distribution of the covariates.

The word “series” here refers to a finite sum. The basis could consist of powers of \(x\), trigonometric functions, spline functions, or other specified functions. The choice of basis determines the set of functions available to the fit; the least-squares calculation chooses a member of that set. The coefficients depend on the particular basis representation, so it is the fitted function—not an isolated coefficient—that describes the resulting regression estimate.

Empirical Least Squares for a Fixed Design

First suppose the covariates \(x_1,\ldots,x_n\) are fixed and the observed responses are \(y_1,\ldots,y_n\). Define the \(n\)-by-\(K\) design matrix \(B\) to have row \(i\) equal to \(\phi(x_i)^\mathsf{T}\), and let \(y=(y_1,\ldots,y_n)^\mathsf{T}\). Least squares minimizes the empirical sum of squared residuals.

When the columns of \(B\) are linearly independent, \(B^\mathsf{T}B\) is invertible and the coefficient vector is unique. The next result derives its formula rather than assuming it.

Proof. For any vector \(h\in\mathbb{R}^K\), expand the objective along the line \(\beta+th\):

At a minimizer, the coefficient of \(t\) must be zero for every direction \(h\); otherwise a sufficiently small \(t\) of the appropriate sign would reduce the objective. Thus \(B^\mathsf{T}(y-B\hat\beta)=0\), or \(B^\mathsf{T}B\hat\beta=B^\mathsf{T}y\). Full column rank makes \(B^\mathsf{T}B\) positive definite: for nonzero \(h\), \(h^\mathsf{T}B^\mathsf{T}Bh=\|Bh\|^2>0\). The normal equations therefore have the displayed unique solution. Substituting that solution into the expansion shows that for every nonzero \(h\), \(Q_n(\hat\beta+h)=Q_n(\hat\beta)+\|Bh\|^2>Q_n(\hat\beta)\). Hence it is the unique minimizer. \(\square\)

If \(B\) is rank deficient, the inverse formula is not available and the coefficient vector need not be unique. In that case, different coefficient vectors can represent the same fitted values on the observed design. Full column rank is therefore a condition for unique empirical coefficients, not a requirement that one can omit when using the displayed inverse formula.

Population Projection Coefficients

For random covariates, there is also a population target: the best approximation to the regression function using the chosen basis, where “best” is measured on average over the covariate distribution. Let \(X\) be a random covariate and let \(m(x)=E[Y\mid X=x]\) be a version of the regression function. Assume \(E[m(X)^2]<\infty\) and \(E[\phi_j(X)^2]<\infty\) for every \(j\). Define the population criterion

Positive definiteness of \(G\) means that no nonzero linear combination of the basis functions is zero with probability one under the distribution of \(X\). The population coefficient equations follow from minimization of the expected squared approximation error.

Derivation. For any \(h\in\mathbb{R}^K\), expansion gives

The expectations exist by the assumed second moments and the Cauchy–Schwarz inequality. At a minimizer, the linear term must vanish in every direction, so \(G\beta^*=q\). Since \(G\) is positive definite, the solution is unique. Conversely, for this solution the expansion reduces to \(Q(\beta^*+h)=Q(\beta^*)+h^\mathsf{T}Gh\), which is strictly larger for every nonzero \(h\). This proves both minimization and uniqueness. \(\square\)

If \(E[Y^2]<\infty\), the same coefficient vector also minimizes \(E[\{Y-\phi(X)^\mathsf{T}\beta\}^2]\). Indeed, its directional equations use \(E[\phi(X)Y]\), and the defining property of conditional expectation gives \(E[\phi(X)Y]=E[\phi(X)m(X)]\). The empirical and population equations look similar, but they use different quantities: the former uses sums over the observed sample, while the latter uses expectations under the distribution of \(X\).

Random-Design Empirical Coefficients

With random design, the observed covariates \(X_1,\ldots,X_n\) generate a random design matrix \(B\), whose \(i\)th row is \(\phi(X_i)^\mathsf{T}\). Conditional on the observed covariates, the empirical least-squares calculation is exactly the fixed-design calculation. When the realized matrix has full column rank,

This random vector is the empirical coefficient estimate. Its matrix and vector are sample averages, not population expectations. Even when the population coefficient vector is well-defined, a finite sample generally produces different coefficients. Sample rank also matters: a population matrix \(G\) can be positive definite while a particular realized sample matrix is singular.

Why the Distinction Matters

A basis expansion specifies the shape of the candidate functions, while the coefficient calculation specifies how the data or population distribution selects among them. Keeping those roles separate helps interpret results correctly: a coefficient is tied to a particular basis, and an empirical estimate is tied to a particular sample. In random-design analysis, the population coefficients describe the target approximation, whereas the empirical coefficients are the quantities calculated from observed data.

Common mistake: Replacing the sample averages in the empirical normal equations with population expectations and then treating the resulting coefficients as the least-squares fit to the observed data. Those are not the same calculation. The correction is to identify the target first: use \(B^\mathsf{T}B\) and \(B^\mathsf{T}Y\) for the realized sample, and \(E[\phi(X)\phi(X)^\mathsf{T}]\) and \(E[\phi(X)m(X)]\) for the population approximation. Equality may occur in a particular sample or special model, but it is not guaranteed by the definitions.