Nodewise Lasso for Approximate Precision Matrices

Why Estimate Precision-Matrix Rows by Regression?

The debiased lasso uses an approximate inverse of the design covariance matrix to correct regularization bias. When the number of covariates \(p\) is large relative to the sample size \(n\), directly inverting the empirical covariance matrix may be impossible or unstable. Nodewise lasso provides another route: estimate one row at a time by regressing each covariate on all the others, using an \(\ell_1\) penalty to encourage sparse regressions.

The key connection is structural. If a covariate can be well predicted by a small number of the remaining covariates, its regression residual provides a direction that is nearly orthogonal to those other covariates. Rescaling that direction gives an approximate inverse-covariance row. This construction is useful only when the nodewise regressions are sufficiently sparse and the design supports their estimation.

Population Regression and the Precision Matrix

Let \(X\) be a centered \(n \times p\) design matrix, with column \(x_j\) containing observations on covariate \(j\). Write \(\Sigma\) for the population covariance matrix of the covariates, and suppose it is nonsingular. For each \(j\), define the population linear regression of covariate \(j\) on all the others. Its coefficient vector is \(\gamma_j = \Sigma_{-j,-j}^{-1}\Sigma_{-j,j}\), where \(-j\) means all indices except \(j\). Its residual variance is \(\tau_j^2 = \Sigma_{jj}-\Sigma_{j,-j}\gamma_j\).

To see the connection, place a 1 in position \(j\) and the entries of \(-\gamma_j\) in the other positions, forming a vector \(v_j\). The population regression equations imply that \(\Sigma v_j\) is zero outside position \(j\): for the other covariates, the residual is uncorrelated with each regressor. The \(j\)th entry of \(\Sigma v_j\) is the residual variance \(\tau_j^2\). Therefore \(\Sigma(v_j/\tau_j^2)=e_j\), where \(e_j\) is 1 in position \(j\) and 0 elsewhere. Multiplying by \(\Sigma^{-1}\) shows \(v_j/\tau_j^2=\Omega e_j\), the \(j\)th column of \(\Omega\). Since \(\Omega\) is symmetric, this is also its \(j\)th row, written as a vector.

The Sample Nodewise Lasso

In the sample, estimate each population regression with a lasso regression. Define the empirical covariance matrix as \(\hat\Sigma=X^\top X/n\). For each \(j\), let \(X_{-j}\) denote \(X\) with column \(j\) removed, and solve

Let \(\hat r_j=x_j-X_{-j}\hat\gamma_j\) be the residual vector. A residual-based scale estimate includes a penalty correction:

The resulting estimate of row \(j\) places \(1/\hat\tau_j^2\) in position \(j\), and \(-\hat\gamma_j/\hat\tau_j^2\) in the other positions. Collecting these rows gives an approximate precision-matrix estimate, often denoted \(\hat\Omega\). If the nodewise coefficients are sparse, each row can be estimated without solving a single high-dimensional matrix-inversion problem.

Thus the estimated row has a normalized product with \(x_j\) equal to 1. Its products with the other design columns are controlled by the lasso penalty rather than being exactly zero. This is why the result is called an approximate inverse.

Worked Example: Calculating a Row from Four Observations

Conditions for the Approximation

Nodewise lasso does not make a dense regression estimable just by adding a penalty. Its performance depends on assumptions about the covariate distribution, the design, and the sparsity of the nodewise regressions. In a common high-dimensional regime, the number of covariates may exceed the sample size, so an ordinary inverse of \(\hat\Sigma\) does not exist. The following conditions describe the basic requirements; precise theorems vary in their details.

• Nonsingular population covariance: \(\Sigma\) has eigenvalues bounded away from zero and infinity. This prevents the target precision matrix from being arbitrarily ill-conditioned.
• Controlled design for each regression: the empirical design must satisfy a compatibility or restricted-eigenvalue condition on the sparse directions relevant to each nodewise regression. These conditions prevent distinct sparse coefficient vectors from producing nearly indistinguishable fitted values.
• Sparse or approximately sparse regressions: each \(\gamma_j\) has relatively few important entries. If \(s_j\) denotes its number of nonzero entries, the maximum row sparsity \(s_{\max}=\max_j s_j\) must grow slowly enough relative to \(n\) and \(p\). Approximate sparsity is also possible when the small coefficients have a sufficiently controlled tail.
• Appropriate penalty: \(\lambda_j\) is typically chosen on the order of \(\sqrt{\log(p)/n}\), with constants or scaling adjusted to the noise and covariate scales. This size controls the maximum empirical correlation between a regression residual and the candidate regressors.

For example, standard lasso bounds under suitable design and score conditions give a prediction error of order \(s_j\lambda_j^2\) and an \(\ell_1\) coefficient error of order \(s_j\lambda_j\), up to constants. These rates explain why \(s_j\log(p)/n\) should be small for prediction error to vanish, and why stronger sparsity control can be needed when accurate coefficient or inverse-matrix estimates are required. These are rate guides, not universal guarantees: the exact assumptions and error bounds depend on the design and the target.

Worked Example: A Three-Covariate Nodewise Regression

What the Estimate Does—and Does Not—Guarantee

Nodewise lasso produces one regression-based estimate for each row. Those regressions are solved separately, so the resulting \(\hat\Omega\) need not be exactly symmetric. It is also not guaranteed to be positive definite. Neither property follows merely from the row construction. Some applications symmetrize the estimate, but the choice of symmetrization is an additional step; it does not turn the original regression estimates into exact inverse rows.

Sparsity has a useful interpretation when covariates are jointly Gaussian: a zero off-diagonal entry of the population precision matrix corresponds to conditional independence of the associated pair given the other covariates. More generally, a sparse nodewise regression means that covariate \(j\) is linearly explained by a relatively small subset of the others. Without the relevant distributional assumptions, a zero regression coefficient should not automatically be read as conditional independence.

In debiased-lasso procedures, the estimated rows supply directions used to reduce the regularization bias of the original lasso estimate. Their quality matters because errors in the approximate inverse can affect the correction. Nodewise lasso therefore does not eliminate high-dimensional assumptions: it replaces direct covariance inversion with a collection of sparse regression problems whose errors must still be controlled.

Common Mistakes

• Using only the raw residual variance. The lasso KKT conditions imply the correction \(\lambda_j\|\hat\gamma_j\|_1\). Omitting it generally changes the scale and the resulting row.
• Calling the estimate an exact inverse. The products with non-target columns are controlled by the penalty, not forced to zero. Check the off-diagonal products before describing the row as an exact inverse.
• Assuming sparsity alone is enough. Sparse nodewise regressions still require a design condition and suitable control of empirical score correlations. A small support does not guarantee stable estimation if the regressors are too collinear.
• Assuming the assembled estimate is symmetric or positive definite. Rowwise regression does not impose either constraint. Do not treat those properties as automatic.