Pinsker Shrinkage for Series Regression

From a Smoothness Constraint to Shrinkage

A series regression estimator must balance two sources of error. Retaining many estimated coefficients reduces approximation bias, but each retained coefficient contributes noise. Hard truncation makes a single decision about each frequency: keep it or discard it. Pinsker shrinkage instead attenuates coefficients continuously, with the amount of attenuation determined by the frequency’s contribution to a smoothness constraint.

The derivation is especially transparent in a Gaussian sequence model. There, the observations are noisy versions of unknown coefficients, and the smoothness assumption restricts those coefficients to an ellipsoid. We will find the linear rule with the smallest worst-case squared-error risk over that ellipsoid. Its weights have a simple form: they decrease with the ellipsoid’s frequency weights and become zero beyond a cutoff.

The values \(a_j\) describe how strongly the constraint penalizes each coefficient. In a smoothness ellipsoid, high-frequency coefficients typically have larger \(a_j\), so the constraint allows less energy at high frequencies. We study estimators of the form \(\hat\theta_j=\lambda_jY_j\), where each weight \(\lambda_j\) is fixed rather than chosen from the observed data. Their risk is expected squared Euclidean error, \(E_\theta[\sum_j(\hat\theta_j-\theta_j)^2]\), and their worst-case risk is the supremum of this quantity over \(\mathcal E(Q,a)\).

Worst-Case Risk and the Pinsker Rule

For a fixed coefficient \(\theta_j\), the estimation error separates into shrinkage bias and noise: \[ E_\theta[(\lambda_jY_j-\theta_j)^2] =(1-\lambda_j)^2\theta_j^2+\tau^2\lambda_j^2. \] The first term grows when the weight is made smaller; the second grows when it is made larger. Summing over coordinates gives the full risk. The ellipsoid lets us calculate the largest possible total bias exactly.

To see why, set \(c=\max_j (1-\lambda_j)^2/a_j^2\). Each squared-bias coefficient is at most \(c a_j^2\), so the sum is at most \(cQ\) on the ellipsoid. Equality is attained by putting all allowed ellipsoid energy on a coordinate that maximizes the ratio. This shows why a worst-case risk calculation is governed by the most weakly protected coordinate, rather than by an average over frequencies.

Proof. First, it is enough to consider coordinatewise estimators. For a diagonal sign matrix \(S\), whose diagonal entries are \(1\) or \(-1\), the model and ellipsoid are invariant under \((Y,\theta)\mapsto(SY,S\theta)\). If \(\delta(y)=Ay+b\), define \(\delta_S(y)=S\delta(Sy)\). Its maximum risk equals that of \(\delta\), since the transformed noise has the same distribution and \(S\theta\) ranges over the same ellipsoid. Average \(\delta_S\) over all \(2^p\) sign matrices. Squared error is convex in the estimator, so the averaged estimator has maximum risk no greater than that of \(\delta\). In the average, every off-diagonal entry of \(A\) and every entry of \(b\) cancels; the result is a coordinatewise linear estimator. Thus allowing a general affine linear map cannot improve the minimax risk.

For coordinatewise weights, clipping any \(\lambda_j\) to the interval \([0,1]\) cannot increase either its squared bias coefficient \((1-\lambda_j)^2\) or its variance contribution \(\tau^2\lambda_j^2\). We may therefore restrict to \(0\le\lambda_j\le1\). Put \(t=\max_j(1-\lambda_j)/a_j\). The ellipsoid calculation above gives worst-case risk \(\tau^2\sum_j\lambda_j^2+Qt^2\). For a fixed \(t\), the constraint defining \(t\) requires \(\lambda_j\ge 1-ta_j\), as well as \(\lambda_j\ge0\). Since variance increases with \(\lambda_j\) on \([0,1]\), the smallest variance satisfying these constraints is attained at \(\lambda_j=(1-ta_j)_+\). This proves that minimizing worst-case risk over the weights reduces to minimizing \(R(t)\).

On any interval where the active set \(\{j:ta_j<1\}\) is fixed, differentiation gives \(R'(t)=2Qt-2\tau^2\sum_j a_j(1-ta_j)_+\). At \(t=0\), the right derivative is \(-2\tau^2\sum_j a_j<0\). For \(t\ge 1/\min_j a_j\), all weights are zero and \(R(t)=Qt^2\), which is increasing. Moreover, \(Qt\) increases strictly and \(\tau^2\sum_j a_j(1-ta_j)_+\) decreases continuously. They therefore intersect exactly once, at the minimizer. Substitution gives the stated risk. \(\square\)

Worked Calculations

Interpretation in Series Regression

Suppose a regression function is expanded in an orthonormal basis as \(f=\sum_j\theta_j\phi_j\), and the observed series coefficients satisfy the Gaussian sequence approximation \(Y_j=\theta_j+\tau Z_j\). The Pinsker estimator fits \(\hat f=\sum_j(1-ta_j)_+Y_j\phi_j\). Under orthonormality, squared integrated error for the fitted series equals the sum of squared coefficient errors, so the sequence risk calculation directly describes the regression risk for those coefficients.

For a smoothness constraint with \(a_j\) increasing by frequency, low-frequency weights are close to one and high-frequency weights are smaller. Frequencies for which \(ta_j\ge1\) receive weight zero, while those below the cutoff are continuously attenuated. The value of \(t\) balances worst-case squared bias against the accumulated variance and depends on the noise level, sample size through \(\tau^2\), and the ellipsoid radius and shape. In a nonorthogonal or heteroskedastic setting, coefficient noise need not be independent with a common variance; the displayed rule does not automatically apply without accounting for that covariance structure.

Why the Linear Qualification Matters

The theorem is a minimax statement for affine linear estimators under the specified Gaussian model and ellipsoidal constraint. It does not say that a fixed Pinsker rule is best among every estimator that can depend nonlinearly on the observed coefficients. Nor does it select \(Q\), the smoothness weights \(a_j\), or a tuning parameter from data. The weights are determined by the stated model and constraint. Data-driven risk estimation for series shrinkage is a separate question.

A common mistake is to treat the rule as hard truncation because some weights equal zero. The zero weights occur only once \(ta_j\ge1\); before that point, the coefficient is multiplied by \(1-ta_j\), not retained unchanged. Another mistake is to assert that high-frequency coefficients are always discarded. Whether they are depends on \(t\) and the \(a_j\): as the regression example shows, every coefficient can remain active while still being shrunk by different amounts.