Penalized Prediction Versus Classical Inference

Why Prediction and Coefficient Inference Can Diverge

Penalization can improve prediction by stabilizing estimates, especially when predictors are numerous or strongly related. But a coefficient chosen to optimize a penalized objective is generally not an ordinary maximum-likelihood estimate: the penalty deliberately pulls it away from values favored by the data likelihood. Thus a useful prediction rule need not provide an unbiased estimate of the corresponding population coefficient, and the usual likelihood standard error does not automatically quantify uncertainty about that coefficient.

The distinction is between a penalized target, the population value toward which a penalized procedure converges, and the true parameter in a correctly specified GLM. Inference around the penalized target is not, by itself, inference about the true parameter. The gap matters most when the penalty remains appreciable as the sample size grows or when tuning and variable selection are performed using the same data.

The Penalized Target and Its Bias

Let \(Y_1,\ldots,Y_n\) be independent and identically distributed observations, and let \(\ell_i(\beta)\) be the log-likelihood contribution for observation \(i\), with coefficient vector \(\beta\). For a fixed penalty level \(\rho\geq 0\) and a differentiable penalty \(P(\beta)\), consider the estimator

This scaling makes \(\rho\) the penalty relative to the average log-likelihood. Other software conventions may multiply the likelihood or penalty by \(n\); the substantive comparison is unchanged once the scaling is translated. For example, ridge uses \(P(\beta)=\tfrac12\beta^\mathsf{T}\beta\), often with the intercept excluded from the penalty.

If the likelihood is correctly specified, \(E[s_i(\beta_0)]=0\). But at \(\beta_0\), the penalized population equation has left-hand side \(-\rho\nabla P(\beta_0)\). Unless this penalty gradient is zero, or the penalty vanishes, \(\beta_0\) is not the penalized target. This is the mechanism behind shrinkage bias. The amount and direction of that bias depend on the penalty and its tuning parameter.

Sampling Variance Around a Fixed Penalized Target

For a smooth penalty and a fixed \(\rho\), uncertainty around \(\beta_\rho\) can be derived using the estimating equation. Write

Here \(J\) is the expected negative likelihood Hessian, \(\Omega\) is the covariance of the individual likelihood scores, and \(A\) is the derivative curvature of the penalized population estimating equation, with sign reversed. Suppose the observations are iid; the parameter dimension is fixed; \(\beta_\rho\) is an interior solution; \(P\) and the log-likelihood contributions are twice continuously differentiable near it; the required score second moments are finite; \(A(\beta_\rho)\) is nonsingular; and standard uniform laws of large numbers, a central limit theorem, and a Taylor expansion with a negligible remainder hold. Also suppose the estimator converges in probability to this solution.

Proof. The estimator satisfies the sample penalized score equation. Expand it about \(\beta_\rho\), whose population equation is zero. The derivative of the sample equation converges in probability to \(-A(\beta_\rho)\). Consequently,

The first two terms together have mean zero by the target equation. The central limit theorem gives \(n^{-1/2}\sum_i s_i(\beta_\rho)\xrightarrow{d}N(0,\Omega(\beta_\rho))\). Nonsingularity of \(A\), the Taylor expansion, and consistency yield the first-order representation

Applying the central limit theorem and the continuous mapping theorem gives the stated covariance. The penalty changes the curvature through \(A\), but it contributes no random score term when \(\rho\) is fixed. \(\square\)

The theorem is about a smooth, fixed-penalty estimator and its target. It does not say that a usual likelihood standard error is valid for inference about \(\beta_0\). Nor does it directly cover a nonsmooth penalty such as the lasso at a coefficient equal to zero: there the derivative used in this argument may not exist, and the limiting distribution can behave differently.

Information Equality Must Be Evaluated at the Right Parameter

Under a correctly specified, regular likelihood, the information equality states that \(\Omega(\beta_0)=J(\beta_0)\) at the true parameter \(\beta_0\). It does not license replacing \(\Omega(\beta_\rho)\) by \(J(\beta_\rho)\) when the penalized target differs from the true parameter. At that target, the likelihood score may have nonzero expectation, and its variance need not equal the expected negative Hessian.

Even in a correctly specified model, the theorem’s sandwich covariance must therefore use \(\Omega\) evaluated at \(\beta_\rho\), unless the target coincides with \(\beta_0\) and the regularity conditions for information equality hold there. A positive penalty also contributes \(\rho\nabla^2P(\beta_\rho)\) to \(A\); this curvature contribution is not itself a source of random score variation.

Why Ordinary Likelihood Standard Errors and Tests Can Fail

For an unpenalized maximum-likelihood estimator in a correctly specified regular model, the usual information-based covariance is justified by the likelihood score equation, centering at the true parameter, and information equality. Penalization changes these ingredients. The estimator solves a different equation, is generally centered asymptotically at \(\beta_\rho\), and has derivative curvature \(A\), not just \(J\). The random score variation is \(\Omega\). Consequently, simply taking the inverse likelihood Hessian—or adding penalty curvature to that Hessian without accounting for score variation—does not generally produce the covariance in the theorem.

A Wald interval centered at \(\hat\beta_\rho\) and based on a valid sandwich standard error can describe uncertainty around \(\beta_\rho\) under the theorem’s assumptions. It still need not cover \(\beta_0\) at its advertised rate or frequency when the target has nonvanishing bias. A test of a true coefficient also cannot be justified merely by applying the usual maximum-likelihood Wald formula to a penalized coefficient. The estimand, centering, and uncertainty calculation must all match the inferential question.

When the penalty level tends to zero with sample size, classical inference may become available under additional conditions. For a smooth penalty in a fixed-dimensional regular model, the population target’s displacement is often of order \(\rho\) near \(\beta_0\). If that displacement is negligible relative to the \(n^{-1/2}\) sampling scale, and the estimator has the required regularity, the penalty may be asymptotically negligible for inference. This is not automatic: for lasso penalties, tuning selected from the data, or growing-dimensional settings, different arguments and limiting behavior may be required.

Prediction procedures also commonly choose the tuning parameter from the data, for example by cross-validation. The fixed-\(\rho\) derivation above does not include the extra randomness from that choice. Nor does it account for the fact that selected variables may change across samples. Treating a selected set as if it had been specified in advance can understate uncertainty. These cautions do not invalidate penalized prediction; they mean that predictive performance and classical coefficient inference answer different questions.

Common Pitfall

A sound analysis states whether its goal is prediction, inference about a penalized target, or inference about a true model parameter. It reports how tuning was chosen, distinguishes shrinkage bias from sampling variability, and uses an inferential method justified for that goal. A smaller standard error alone does not show that penalization has made coefficient inference more reliable.