The Debiased Lasso Estimator

Why Correct the Lasso Estimate?

In a regression with many candidate predictors, the lasso can estimate a sparse coefficient vector while controlling estimation error. Its penalty also shrinks selected coefficients toward zero. That regularization bias matters when the goal is to estimate a particular coefficient, rather than only to predict well. The debiased lasso keeps the regularized estimate as its starting point, then uses the regression residuals to correct part of that shrinkage.

The correction depends on a direction that acts like the relevant row of the inverse design Gram matrix. When the number of predictors is large, the Gram matrix may not be invertible, or its inverse may be too unstable to use directly. The construction therefore uses an approximate inverse. The central algebra shows what this approximation achieves and what error remains.

Model, Lasso Estimate, and Approximate Inverse

Consider the linear model with $n$ observations and $p$ predictors:

y=X\beta+\varepsilon.

Here $y$ is an $n$-vector of outcomes, $X$ is an $n\times p$ design matrix, $\beta$ is the $p$-vector of population regression coefficients, and $\varepsilon$ is the $n$-vector of errors. Assume the columns of $X$ have been centered and scaled so that each has squared length $n$, and that any intercept has been removed or handled separately. For the algebra below, the model identity is sufficient. For statistical interpretation, a usual condition is that the errors have conditional mean zero given the design, $E(\varepsilon\mid X)=0$. Additional assumptions are needed for uncertainty estimates and asymptotic inference.

Write the sample Gram matrix and the score at a coefficient vector $b$ as

\hat{\Sigma}=\frac{X'X}{n}, \qquad \hat{S}(b)=\frac{X'(y-Xb)}{n}.

Use the lasso estimate $\hat{\beta}$ that minimizes the squared-error loss plus an $\ell_1$ penalty:

\hat{\beta} = \arg\min_{b\in\mathbb{R}^p} \left\{ \frac{1}{2n}\|y-Xb\|_2^2+\lambda\|b\|_1 \right\}, \qquad \lambda>0.

For a target coordinate $j$, let $e_j$ be the $p$-vector with a one in position $j$ and zeros elsewhere. We seek a vector $m_j$ such that $m_j'\hat{\Sigma}$ is close to $e_j'$. In other words, $m_j'$ should approximately undo the action of the Gram matrix in the direction needed to isolate coefficient $j$.

Definition: Given a lasso estimate $\hat{\beta}$ and a vector $m_j$ satisfying $m_j'\hat{\Sigma}\approx e_j'$, the debiased lasso estimator of the $j$th coefficient is $$ \hat{b}_j=\hat{\beta}_j+m_j'\frac{X'(y-X\hat{\beta})}{n}. $$ The second term is the correction: it is the residual score projected along the approximate inverse direction $m_j$.

The construction is useful only if the approximate inverse is accurate enough in the relevant sense. A common requirement is that $\|e_j'-m_j'\hat{\Sigma}\|_\infty$ be small. How to obtain such vectors is a separate design problem; here the focus is on what the estimator does once a suitable $m_j$ has been chosen.

Why the Correction Works: An Exact Decomposition

Substitute the model $y=X\beta+\varepsilon$ into the estimator. The residual score at $\hat{\beta}$ is $\hat{\Sigma}(\beta-\hat{\beta})+X'\varepsilon/n$. Therefore,

\begin{aligned} \hat{b}_j &=\hat{\beta}_j +m_j'\left\{\hat{\Sigma}(\beta-\hat{\beta}) +\frac{X'\varepsilon}{n}\right\}\\ &=\beta_j +m_j'\frac{X'\varepsilon}{n} +\left(e_j'-m_j'\hat{\Sigma}\right)(\hat{\beta}-\beta). \end{aligned}

This is an exact finite-sample identity, not an approximation. The first term is the target coefficient. The second is a noise term. The last term is the remainder: it is the part of the lasso estimation error that the approximate inverse did not remove.

Key result: The debiased estimate has a leading error term $m_j'X'\varepsilon/n$ and a remainder $(e_j'-m_j'\hat{\Sigma})(\hat{\beta}-\beta)$. The correction removes the leading shrinkage bias when the approximate-inverse error is small, but does not automatically eliminate all estimation error.

The remainder admits a direct bound. For any vectors $a$ and $v$, $|a'v|\leq\|a\|_\infty\|v\|_1$. Applying that inequality gives

\left| \left(e_j'-m_j'\hat{\Sigma}\right)(\hat{\beta}-\beta) \right| \leq \left\|e_j'-m_j'\hat{\Sigma}\right\|_\infty \left\|\hat{\beta}-\beta\right\|_1.

This explains why the construction pairs an approximate inverse with a lasso estimate: a small inverse-approximation error multiplied by a controlled $\ell_1$ estimation error can make the remainder small. The lasso oracle inequalities and compatibility conditions provide relevant control of estimation error under sparsity and design assumptions. The identity itself requires neither sparsity nor a probability limit; those assumptions matter when establishing that the remainder is negligible relative to the noise term.

Worked Constructions

Worked Example: Orthogonal Design and a Scalar Correction

Suppose there is one standardized predictor, so that $\hat{\Sigma}=1$. Let the observed score at zero be $X'y/n=1.4$, and set $\lambda=0.3$. Find the lasso estimate and its debiased version for the only coefficient.

In this one-dimensional case, the lasso objective is $(y-Xb)'(y-Xb)/(2n)+0.3|b|$. Its minimizer is the soft-thresholded score. Since $1.4>0.3$, the lasso estimate is

\hat{\beta}=1.4-0.3=1.1.

The inverse Gram matrix is exactly one, so choose $m_1=1$. The residual score at the lasso estimate is

\frac{X'(y-X\hat{\beta})}{n} =\frac{X'y}{n}-\hat{\Sigma}\hat{\beta} =1.4-1.1=0.3.

Thus the correction and the debiased estimate are

m_1\frac{X'(y-X\hat{\beta})}{n}=1(0.3)=0.3, \qquad \hat{b}_1=1.1+0.3=1.4.

Here the correction exactly reverses the lasso shrinkage. This is a special, transparent case: with one standardized predictor, the inverse Gram direction is exact.

Worked Example: Correcting a Target in a Correlated Two-Predictor Design

Suppose a hypothetical standardized design has $\hat{\Sigma}=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}$, the score at zero is $X'y/n=(1.2,0.6)'$, and $\lambda=0.2$. The lasso solution is $\hat{\beta}=(1,0)'$. Verify that it satisfies the lasso optimality conditions, then calculate a debiased estimate for coordinate 1 using $m_1=(1,-0.5)'$.

The lasso score at the proposed estimate is

\frac{X'(y-X\hat{\beta})}{n} = \frac{X'y}{n}-\hat{\Sigma}\hat{\beta} = \begin{pmatrix}1.2\\0.6\end{pmatrix} - \begin{pmatrix}1\\0.5\end{pmatrix} = \begin{pmatrix}0.2\\0.1\end{pmatrix}.

For a nonzero positive coefficient, the lasso score must equal $\lambda$; indeed the first component is $0.2$. For a zero coefficient, its score must have absolute value no greater than $\lambda$; here $|0.1|\leq0.2$. Thus the proposed estimate satisfies the lasso KKT conditions.

The chosen vector is approximate rather than exact. Its product with the Gram matrix is

m_1'\hat{\Sigma} = \begin{pmatrix}1&-0.5\end{pmatrix} \begin{pmatrix}1&0.5\\0.5&1\end{pmatrix} = \begin{pmatrix}0.75&0\end{pmatrix}.

The approximate-inverse error in the target direction is therefore $e_1'-m_1'\hat{\Sigma}=(0.25,0)$. The residual-score correction is

m_1'\frac{X'(y-X\hat{\beta})}{n} = \begin{pmatrix}1&-0.5\end{pmatrix} \begin{pmatrix}0.2\\0.1\end{pmatrix} =0.2-0.05=0.15.

Consequently, the debiased estimate is $\hat{b}_1=1+0.15=1.15$. The correction moves the lasso estimate for the first coefficient upward by $0.15$. It does not establish that $1.15$ equals the unknown population coefficient; remaining approximation error and sampling noise are captured by the decomposition.

Worked Example: Quantifying the Remainder

Consider a hypothetical target coefficient with $\beta_j=0.80$. Suppose the lasso estimation error is $\hat{\beta}-\beta=(0.10,-0.08,0.06)'$, the approximate-inverse error is $e_j'-m_j'\hat{\Sigma}=(0.04,-0.02,0.01)$, and the noise term $m_j'X'\varepsilon/n$ is $0.015$. Calculate the remainder, its norm bound, and the resulting debiased estimate.

The exact remainder is the inner product

$$ (0.04)(0.10)+(-0.02)(-0.08)+(0.01)(0.06) =0.004+0.0016+0.0006=0.0062. $$

The bound uses the largest absolute component of the approximate-inverse error and the sum of the absolute lasso errors:

\|e_j'-m_j'\hat{\Sigma}\|_\infty \|\hat{\beta}-\beta\|_1 = 0.04(0.10+0.08+0.06) =0.04(0.24)=0.0096.

As required, the absolute remainder $0.0062$ is no greater than its bound $0.0096$. The decomposition then gives

\hat{b}_j=0.80+0.015+0.0062=0.8212.

The difference from the target coefficient is $0.0212$: $0.015$ from the noise term and $0.0062$ from the remainder. This calculation illustrates why a small approximate-inverse error is valuable, but also why the correction does not make the remainder identically zero.

What the Estimator Does—and Does Not—Provide

The debiased lasso is a construction for a target coefficient, not a claim that regularization bias disappears under every design. A useful approximate inverse must make $m_j'\hat{\Sigma}$ sufficiently close to $e_j'$, while the lasso estimate must have suitable estimation error. Their product controls the remainder. If predictors are highly dependent or the approximate inverse is poor, the correction may leave a substantial remainder.

Under homoskedastic errors with variance $\sigma^2$, conditional on $X$, the variance of the leading noise term is

\operatorname{Var}\left(m_j'\frac{X'\varepsilon}{n}\,\middle|\,X\right) = \frac{\sigma^2}{n}m_j'\hat{\Sigma}m_j.

This follows by writing the term as $m_j'X'\varepsilon/n$, applying the conditional covariance $\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I_n$, and simplifying. It describes the leading noise component, not automatically the full finite-sample uncertainty of $\hat{b}_j$. Valid inference requires conditions that control the remainder and justify an appropriate approximation to the estimator's distribution.

This distinction is important: bias correction is not, by itself, a confidence interval or a guarantee of valid inference. The estimator's algebra explains how regularization bias is addressed; distributional results require their own assumptions and justification.

Common Pitfall

Wrong reasoning: “The correction uses residuals, so it removes all lasso bias and makes the estimate unbiased.”
Correction: The exact decomposition contains both a noise term and a remainder. The remainder is bounded by the approximate-inverse error times the $\ell_1$ lasso estimation error; it may be small under suitable conditions, but it is not automatically zero.
Wrong reasoning: “Any vector $m_j$ will work because the lasso residual score is available.”
Correction: The vector must approximately invert the Gram matrix in the target direction. If $m_j'\hat{\Sigma}$ is far from $e_j'$, the correction need not isolate coefficient $j$.

Key takeaway: For a chosen coefficient, add the residual score projected along an approximate inverse Gram direction. The resulting estimate equals the target coefficient plus a leading noise term and a remainder whose size is controlled by inverse-approximation and lasso-estimation errors.

Check Your Understanding

Use the estimator definition and its exact decomposition to answer the following.

In the orthogonal, one-predictor example, why did the correction equal the difference between the unpenalized score and the lasso estimate?
For a target coordinate $j$, what does it mean for $m_j$ to approximately invert the Gram matrix in the relevant direction?
Write the exact decomposition of $\hat{b}_j-\beta_j$ and identify its noise and remainder terms.
If the approximate-inverse error has infinity norm $0.03$ and the lasso estimation error has $\ell_1$ norm $0.5$, what upper bound does the norm inequality give for the absolute remainder?
Why does constructing a debiased estimate alone not establish that a confidence interval for the target coefficient is valid?

What You'll Learn