Orthonormal Series Density Estimation

From Basis Functions to a Density Estimate

A density can be represented through coefficients of suitable functions, much as a vector can be represented by its coordinates in a basis. In density estimation, the coefficients are not known: they are expectations under the unknown distribution. A finite orthonormal-series estimator replaces each such expectation with its sample average and retains only a finite number of terms.

This construction has a useful feature and a limitation. Including the constant basis function makes the resulting estimate integrate to one. But a finite sum of basis functions need not be nonnegative everywhere, so having total integral one does not by itself make the estimate a density.

Definition: Let $I=[a,b]$ be an interval of finite length $L=b-a>0$. Let $X_1,\ldots,X_n$ be independent and identically distributed observations with density $f$ supported on $I$. Suppose $\{\phi_j:j=0,1,\ldots\}$ is an orthonormal system in $L^2(I)$ with respect to Lebesgue measure, with $$ \int_I \phi_j(x)\phi_k(x)\,dx = \begin{cases} 1,&j=k,\\ 0,&j\ne k, \end{cases} \qquad \phi_0(x)=\frac{1}{\sqrt{L}}. $$ For a finite truncation level $K\geq 0$, define the empirical coefficient and the finite orthonormal-series estimator by $$ \hat\beta_j=\frac{1}{n}\sum_{i=1}^n \phi_j(X_i), \qquad \hat f_K(x)=\sum_{j=0}^{K}\hat\beta_j\phi_j(x). $$

The density $f$ need not be known to calculate $\hat\beta_j$: each coefficient is obtained by evaluating a known basis function at the observations and averaging. For the coefficient interpretation as an integral against $f$, assume $E[|\phi_j(X_1)|]<\infty$ for the terms being used. In particular, this condition holds when the basis functions are bounded. If $f\in L^2(I)$, the corresponding population coefficient is the usual inner product $\int_I f(x)\phi_j(x)\,dx$.

The chosen constant function is important because it is orthogonal to every other basis function. Thus, for $j\geq 1$, $$ \int_I \phi_j(x)\,dx =\sqrt{L}\int_I \phi_j(x)\phi_0(x)\,dx =0. $$ Also, every observation contributes the same value to the constant coefficient: $$ \hat\beta_0=\frac{1}{n}\sum_{i=1}^n\frac{1}{\sqrt{L}} =\frac{1}{\sqrt{L}}. $$ Consequently, the constant term in the estimator is $1/L$, the uniform density on $I$. It is not an estimated quantity; it is fixed by the requirement that a density have total mass one.

Basic Properties of the Empirical Coefficients

Result: If $E[|\phi_j(X_1)|]<\infty$, then the empirical coefficient is unbiased for the population coefficient: $$ E[\hat\beta_j]=E[\phi_j(X_1)]=\int_I \phi_j(x)f(x)\,dx. $$ For the constant function specified above, $\hat\beta_0=1/\sqrt{L}$ exactly for every sample.

To verify the expectation, use linearity of expectation for the finite sum and the common distribution of the observations: $$ E[\hat\beta_j] =E\left[\frac{1}{n}\sum_{i=1}^n\phi_j(X_i)\right] =\frac{1}{n}\sum_{i=1}^n E[\phi_j(X_i)] =E[\phi_j(X_1)]. $$ The assumed absolute integrability makes this expectation well-defined. Since $X_1$ has density $f$, the definition of expectation gives $E[\phi_j(X_1)]=\int_I\phi_j(x)f(x)\,dx$. For $j=0$, substitute $\phi_0(X_i)=1/\sqrt L$ directly into the sample average to obtain the stated exact value. $\square$

Theorem: For every sample and every finite $K\geq 0$, the estimator that includes $\phi_0=1/\sqrt L$ integrates to one: $$ \int_I \hat f_K(x)\,dx=1. $$

Proof. Each $\phi_j\in L^2(I)$ is integrable on $I$, since the Cauchy–Schwarz inequality gives $\int_I|\phi_j(x)|\,dx\leq\sqrt L\,(\int_I\phi_j(x)^2\,dx)^{1/2}=\sqrt L$. Thus the finite sum can be integrated term by term. Its constant term has integral $\hat\beta_0\int_I\phi_0(x)\,dx=(1/\sqrt L)(\sqrt L)=1$. For every $j=1,\ldots,K$, orthogonality to $\phi_0$ gives $\int_I\phi_j(x)\,dx=0$, so all remaining terms integrate to zero. Therefore $\int_I\hat f_K(x)\,dx=1$. $\square$

This result is about total integral, not sign. Orthogonality controls integrals and inner products; it does not require a finite sum to stay above zero at every point. The following examples make both features visible.

Worked Examples

Worked Example: The Constant Term on a Non-Unit Interval

Take $I=[0,2]$, so $L=2$, and retain only the constant basis function. For any observations $X_1,\ldots,X_n$ in this interval, $\phi_0(x)=1/\sqrt 2$. Calculate the estimate and check its integral.

For each observation, $\phi_0(X_i)=1/\sqrt 2$, so $$ \hat\beta_0=\frac{1}{n}\sum_{i=1}^n\frac{1}{\sqrt 2} =\frac{1}{\sqrt 2}. $$ The $K=0$ estimate is therefore $$ \hat f_0(x)=\hat\beta_0\phi_0(x) =\frac{1}{\sqrt 2}\frac{1}{\sqrt 2} =\frac{1}{2},\qquad 0\leq x\leq 2. $$ Its integral is $\int_0^2(1/2)\,dx=1$. At this truncation level the estimate is simply the uniform density on the interval, irrespective of the sample values.

Worked Example: A Fourier Estimate That Is Negative

On $I=[0,1]$, use the first cosine basis function after the constant: $$ \phi_0(x)=1,\qquad \phi_1(x)=\sqrt{2}\cos(2\pi x). $$ With a single observation $X_1=0$ and truncation level $K=1$, calculate the estimator and find a point where it is negative.

The empirical coefficients are $$ \hat\beta_0=1,\qquad \hat\beta_1=\phi_1(0)=\sqrt{2}\cos(0)=\sqrt{2}. $$ Thus $$ \hat f_1(x)=1+2\cos(2\pi x). $$ At $x=1/2$, $\cos(\pi)=-1$, giving $\hat f_1(1/2)=1-2=-1$. The estimate is negative there. Nonetheless, $\int_0^1\hat f_1(x)\,dx=1$, because $\int_0^1\cos(2\pi x)\,dx=0$. This example shows explicitly that unit integral does not ensure nonnegativity.

Worked Example: Two Observations and Two Fourier Terms

Still on $[0,1]$, use the constant function and the first sine and cosine functions, $$ \phi_0(x)=1,\qquad \phi_c(x)=\sqrt{2}\cos(2\pi x),\qquad \phi_s(x)=\sqrt{2}\sin(2\pi x). $$ For observations $X_1=0$ and $X_2=1/4$, calculate the finite estimate and verify both its integral and a negative value.

The cosine values at the observations are $\sqrt{2}$ and $0$; hence the empirical cosine coefficient is $(\sqrt{2}+0)/2=1/\sqrt{2}$. The sine values are $0$ and $\sqrt{2}$, so the empirical sine coefficient is also $1/\sqrt{2}$. The constant coefficient is $1$. Therefore $$ \hat f(x)=1+\frac{1}{\sqrt{2}}\phi_c(x)+\frac{1}{\sqrt{2}}\phi_s(x) =1+\cos(2\pi x)+\sin(2\pi x). $$ At $x=5/8$, the angle is $2\pi x=5\pi/4$. Both sine and cosine equal $-\sqrt{2}/2$, so $$ \hat f(5/8)=1-\frac{\sqrt{2}}{2}-\frac{\sqrt{2}}{2} =1-\sqrt{2}<0, $$ because $\sqrt{2}>1$. The sine and cosine terms each integrate to zero over $[0,1]$, so the estimate still integrates to one.

Why Truncation Can Produce Negative Values

A finite series combines basis functions with data-dependent coefficients. Orthonormality says that different basis functions have zero inner product; it does not impose pointwise bounds on their weighted sum. In the Fourier examples, the oscillating terms have zero integral but can be strongly negative at particular locations. The fixed constant term contributes the correct total mass, but it may not be large enough to offset those local negative contributions.

This is a structural issue, not an arithmetic error. A density must be nonnegative almost everywhere as well as integrate to one. Therefore a finite orthonormal-series estimate that takes negative values is a signed function with unit integral, not a density in the strict sense. Replacing negative values by zero would change its integral in general; that operation does not preserve the property proved above automatically.

Key takeaway: Empirical basis coefficients are sample averages of basis evaluations. Including the normalized constant function makes a finite series integrate to one, but truncation does not guarantee nonnegativity. Check both properties separately.

Common Mistake

Wrong reasoning: “The estimate integrates to one, so it is a density.” This uses only the total-mass condition and ignores the sign condition.
Correction: A density must be nonnegative almost everywhere and have integral one. The orthonormal-series construction with its constant term guarantees the second property, but the Fourier examples show that the first can fail.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

For an interval of length $L$, what is the coefficient of the constant basis function in every sample, and what constant function does its term contribute to the estimate?
Why does orthogonality to $\phi_0$ imply that each $\phi_j$, $j\geq1$, has integral zero?
For the Fourier estimate $1+2\cos(2\pi x)$, give a point where the estimate is negative and calculate its value.
Explain why an estimate can integrate to one and still fail to be a density.
What condition on $\phi_j(X_1)$ justifies treating its expectation as a finite population coefficient?