Orthogonal Function Coefficients as Data Projections

From Sampled Functions to Coefficients

Suppose a signal has been recorded at a finite list of times, and we want to describe its sampled values using familiar functions such as a constant, a sine, and a cosine. Each function becomes a column: evaluate it at every sample time, then place the results in a vector. If these columns are orthogonal, each coefficient can be found separately with a dot product. The calculation is a finite-data counterpart of the Fourier-series idea that a function can be described through its components along orthogonal sine and cosine functions.

Work over \(\mathbb{R}\). For columns \(\mathbf{u},\mathbf{v}\in\mathbb{R}^m\), their dot product is \(\mathbf{u}^{\top}\mathbf{v}\), and their squared length is \(\|\mathbf{u}\|^2=\mathbf{u}^{\top}\mathbf{u}\). The columns are orthogonal when their dot product is zero. A data vector \(\mathbf{y}\in\mathbb{R}^m\) records one value at each of the \(m\) sample times. A model using columns \(\mathbf{q}_1,\dots,\mathbf{q}_p\) has fitted vector \(c_1\mathbf{q}_1+\cdots+c_p\mathbf{q}_p\), where the real numbers \(c_j\) are its coefficients.

The denominator matters unless the column has squared length \(1\). A column with a larger length can have a larger dot product with the data even when its coefficient is unchanged. Dividing by its squared length corrects for that scaling. In Fourier-series language, a dot product is replaced by a finite sum over sampled times, while the denominator accounts for the sampled function’s length. The exact coefficients depend on which times were sampled and which columns were included; they need not equal coefficients obtained from a continuous function or a different sampling scheme.

A Four-Point Sine-and-Cosine Grid

Take four equally spaced sample times \(t=0,\tfrac14,\tfrac12,\tfrac34\), and evaluate the constant function, \(\cos(2\pi t)\), and \(\sin(2\pi t)\). Their sampled columns are

Direct dot products give \(\mathbf{u}^{\top}\mathbf{c}=0\), \(\mathbf{u}^{\top}\mathbf{s}=0\), and \(\mathbf{c}^{\top}\mathbf{s}=0\). Their squared lengths are \(\mathbf{u}^{\top}\mathbf{u}=4\), \(\mathbf{c}^{\top}\mathbf{c}=2\), and \(\mathbf{s}^{\top}\mathbf{s}=2\). Thus these three columns meet the definition, but they are not all unit length. The constant coefficient is an average over the four entries; the sine and cosine coefficients have denominator \(2\), not \(4\).

This calculation makes the Fourier analogy concrete: the constant, cosine, and sine columns each receive a coefficient measuring the data’s component in that direction. The fitted vector uses only those components. Any part of the data perpendicular to all three columns remains in the residual.

Why the Formula Gives the Closest Fit

Proof. Set \(\mathbf{r}=\mathbf{y}-\widehat{\mathbf{y}}\). For each \(k\), expand its dot product with \(\mathbf{q}_k\):

The second equality uses pairwise orthogonality to remove all terms with \(j\ne k\); the last uses the coefficient formula. Now take any other \(\mathbf{z}\) in the span. It has the form \(\mathbf{z}=\sum_{j=1}^{p}d_j\mathbf{q}_j\) for some scalars \(d_j\). Then \(\mathbf{y}-\mathbf{z}=\mathbf{r}+\sum_{j=1}^{p}(c_j-d_j)\mathbf{q}_j\). The residual is orthogonal to each column, so it is orthogonal to their linear combination. Expanding the squared length therefore gives

Each \(\|\mathbf{q}_j\|^2\) is positive because \(\mathbf{q}_j\ne\mathbf{0}\). Equality holds only if every \(c_j-d_j=0\), so the minimizing coefficient list is unique. Thus \(\widehat{\mathbf{y}}\) is the unique closest vector in the span, and its residual is orthogonal to each column. \(\square\)

This proof explains why the coefficient formula is more than a convenient calculation. Orthogonality separates the squared error into a residual part and a nonnegative part measuring how far an alternative coefficient list moves from the formula’s coefficients. No other choice can improve the fit.

Reading the Coefficients and the Projection Matrix

For a matrix description, place the orthogonal columns into \(\mathbf{Q}=\begin{bmatrix}\mathbf{q}_1&\cdots&\mathbf{q}_p\end{bmatrix}\), and let \(\mathbf{D}\) be the diagonal matrix whose \(j\)th diagonal entry is \(\mathbf{q}_j^{\top}\mathbf{q}_j\). The coefficient vector is \(\mathbf{D}^{-1}\mathbf{Q}^{\top}\mathbf{y}\), and the fitted vector is \(\mathbf{P}\mathbf{y}\), where \(\mathbf{P}=\mathbf{Q}\mathbf{D}^{-1}\mathbf{Q}^{\top}\). The inverse exists because every diagonal entry of \(\mathbf{D}\) is positive.

Proof. By the coefficient formula, \(\mathbf{P}\mathbf{y}=\mathbf{Q}\mathbf{D}^{-1}\mathbf{Q}^{\top}\mathbf{y}=\sum_j(\mathbf{q}_j^{\top}\mathbf{y})/(\mathbf{q}_j^{\top}\mathbf{q}_j)\,\mathbf{q}_j\). The theorem just proved shows this is the projection. Since \(\mathbf{D}^{-1}\) is diagonal, it is symmetric; thus \(\mathbf{P}^{\top}=\mathbf{Q}\mathbf{D}^{-1}\mathbf{Q}^{\top}=\mathbf{P}\). Orthogonality gives \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{D}\), so

Therefore applying \(\mathbf{P}\) twice gives the same result as applying it once. The first application has already placed the vector in the column span, so a second projection does not change it. \(\square\)

Common Pitfall: Forgetting the Column Length

For the first example, the dot product of the constant column with the data is \(\mathbf{u}^{\top}\mathbf{y}=8\). A tempting but incorrect calculation is to call \(8\) the constant coefficient. That would contribute \(8\mathbf{u}\), whose entries are all \(8\); it is not the projection of the data onto the constant column. The correct coefficient is \(8/(\mathbf{u}^{\top}\mathbf{u})=8/4=2\), so the constant component is \(2\mathbf{u}\). The denominator is essential because the sampled constant column has squared length \(4\), not \(1\).

A second caution is that the shortcut requires orthogonal columns. If sampled sine and cosine columns are not orthogonal on the chosen sample times, the separate dot-product formula does not generally give the closest fit. In that case, the columns interact in the coefficient equations, and the orthogonal formula cannot be used as though their cross-products were zero. Check the sampled dot products before applying it. Also check that no included column is zero: its squared length would be zero, making the formula undefined.

Orthogonal sampled columns are useful because they make coefficient interpretation direct: each coefficient records one component of the data, adjusted for the column’s sampled length. The projection viewpoint also makes the residual meaningful: it contains exactly the part perpendicular to the model columns. For periodic data, this gives a finite-sample analogue of Fourier analysis while keeping the computation grounded in vectors, dot products, and the actual observations.