Worked Example: Tuning And Comparing Glasso With Nodewise Lasso

Why Tune and Compare Both Procedures?

Graphical lasso and nodewise lasso use different fitting targets. Graphical lasso estimates one symmetric precision matrix by penalizing its off-diagonal entries. Nodewise lasso instead fits a separate penalized regression for each variable; the resulting directed selections need not agree across the two regressions for an edge. A comparison therefore involves more than listing which edges appear: it requires a stated tuning rule, an explicit symmetrization rule, and a common criterion for evaluating the fitted models.

This worked example follows one simulated, centered, three-variable Gaussian data set. Its training sample has $n=120$ observations, and a separate held-out sample has $m=60$. The example emphasizes the analysis choices and arithmetic. In an actual study, the reported fits and scores would be computed from the observed training and held-out samples, not selected to match a preferred graph.

Objectives and Tuning Rules

Definition: Given a training sample covariance matrix $S$, graphical lasso at penalty $\lambda\geq 0$ estimates a positive-definite precision matrix by minimizing $$ -\log\det(\Theta)+\operatorname{tr}(S\Theta)+\lambda\sum_{j\ne k}|\Theta_{jk}|. $$ For nodewise lasso, the regression for variable $j$ minimizes $$ \frac{1}{2n}\sum_{i=1}^n\left(X_{ij}-\sum_{k\ne j}\beta_{jk}X_{ik}\right)^2+ \lambda\sum_{k\ne j}|\beta_{jk}|. $$ Here $\Theta$ denotes a precision matrix and $\beta_{jk}$ is the coefficient for predictor $k$ in the regression with response $j$. The displayed conventions use the same symbol $\lambda$ for convenience; their numerical penalty scales need not be comparable.

For each method, choose a penalty from a prespecified grid using the same five training-only cross-validation folds. In each fold, fit on four parts and evaluate the held-out Gaussian log-likelihood on the remaining part. Sum the fold log-likelihoods, or average them using the same normalization for every candidate within a method. Do not use the final held-out sample to choose a penalty: doing so would make the reported held-out performance optimistic.

The tuning grid and cross-validation scores for this simulated run are shown below. Scores are mean log-likelihoods per observation, with the common Gaussian constant $-\tfrac{p}{2}\log(2\pi)$ omitted. Larger values are better. The two penalty grids are chosen separately because the objectives have different scaling.

Method	Penalty	Cross-validated score
Graphical lasso	0.10	-1.62
Graphical lasso	0.15	-1.55
Graphical lasso	0.20	-1.58
Nodewise lasso	0.05	-1.60
Nodewise lasso	0.10	-1.52
Nodewise lasso	0.15	-1.56

Thus the selected penalties are $0.15$ for graphical lasso and $0.10$ for nodewise lasso. The numerical values are not evidence that nodewise lasso is better: cross-validation has selected a candidate within each procedure, while a separate held-out comparison is still to come.

Worked Example: Selecting Penalties From Cross-Validation

For graphical lasso, the candidate scores are $-1.62,-1.55,-1.58$. Since $-1.55$ is the largest, select $\lambda=0.15$. For nodewise lasso, $-1.52$ is the largest of $-1.60,-1.52,-1.56$, so select $\lambda=0.10$.

The penalty values should not be compared as if $0.15$ imposes more regularization than $0.10$. Graphical lasso penalizes entries of a precision matrix, while nodewise lasso penalizes regression coefficients in separate regressions. Their coefficient scales and objective normalizations differ. The relevant comparisons are each method's tuning performance over its own grid, the resulting edge sets, and performance under a common evaluation criterion.

Nodewise Selections and Symmetrization

At the selected nodewise penalty, suppose the three fitted regressions give the following directed selections. An entry in row $j$, column $k$ indicates whether predictor $k$ is selected in the regression with response $j$.

Response regression	Selected predictors
1	2, 3
2	1, 3
3	1

The direction matters at this stage: regression 2 selects variable 3, but regression 3 does not select variable 2. The union rule declares an undirected edge present if at least one of its two directed regressions selected it. The intersection rule declares an edge present only if both directed regressions selected it.

Result: For a pair $j,k$, the union rule selects $\{j,k\}$ when $\hat\beta_{jk}\ne 0$ or $\hat\beta_{kj}\ne 0$; the intersection rule selects it when both coefficients are nonzero. Consequently, the intersection edge set is always a subset of the union edge set.

For this run, both regressions select edge $\{1,2\}$, and both select edge $\{1,3\}$. Only regression 2 selects edge $\{2,3\}$. Therefore the intersection graph has edges $\{1,2\}$ and $\{1,3\}$, whereas the union graph also contains $\{2,3\}$.

Worked Example: Applying Both Symmetrization Rules

Write the directed selection indicators as $A_{jk}=1$ if regression $j$ selects $k$, and $A_{jk}=0$ otherwise. The directed selections above give $A_{12}=A_{21}=1$, $A_{13}=A_{31}=1$, $A_{23}=1$, and $A_{32}=0$.

For $\{1,2\}$, the union decision is $1\lor1=1$, and the intersection decision is $1\land1=1$. For $\{1,3\}$, the decisions are also $1\lor1=1$ and $1\land1=1$. For $\{2,3\}$, they are $1\lor0=1$ and $1\land0=0$. Thus the union selects three edges and the intersection selects two.

The subset claim follows directly: if $A_{jk}\land A_{kj}=1$, then both indicators equal 1, so $A_{jk}\lor A_{kj}=1$. Hence every intersection edge is also a union edge. This is a statement about the rules, not a guarantee that either edge set equals the true graph. $\square$

Held-Out Likelihood on a Common Scale

A symmetric precision estimate can be evaluated on held-out observations using their covariance matrix. For a zero-mean $p$-variate Gaussian model with positive-definite precision matrix $\Theta$, the log density at $x$, excluding its common constant, is $\tfrac12\log\det(\Theta)-\tfrac12x^\mathsf{T}\Theta x$. Averaging over held-out observations with sample covariance $H$ replaces the quadratic term's average by $\operatorname{tr}(H\Theta)$.

Result: For centered held-out observations with sample covariance $H=m^{-1}\sum_{i=1}^{m}x_ix_i^\mathsf{T}$, the average Gaussian log-likelihood, omitting the common constant, is $$ \bar\ell(\Theta;H)=\frac12\{\log\det(\Theta)-\operatorname{tr}(H\Theta)\}. $$ This formula requires $\Theta$ to be positive definite. If the mean is estimated rather than known to be zero, center the held-out observations using the mean specified by the evaluation procedure and use their corresponding covariance.

To verify the formula, sum the log densities over the $m$ held-out observations and divide by $m$. The quadratic terms satisfy $$ \frac{1}{m}\sum_{i=1}^m x_i^\mathsf{T}\Theta x_i =\frac{1}{m}\sum_{i=1}^m\operatorname{tr}(x_i^\mathsf{T}\Theta x_i) =\operatorname{tr}\left(\Theta\frac{1}{m}\sum_{i=1}^m x_ix_i^\mathsf{T}\right) =\operatorname{tr}(\Theta H). $$ The determinant term is the same for every observation. Substitution gives the stated expression. $\square$

In this example, the held-out covariance is $$ H=\begin{pmatrix}1&0.4&0.1\\0.4&1&0.2\\0.1&0.2&1\end{pmatrix}. $$ The simulated fitting run returned the following graphical-lasso estimate at its selected penalty: $$ \hat\Theta_{\mathrm{G}}= \begin{pmatrix}1.2&-0.3&0\\-0.3&1.1&-0.2\\0&-0.2&0.9\end{pmatrix}. $$ Its nonzero off-diagonal entries correspond to edges $\{1,2\}$ and $\{2,3\}$. The determinant is $$ \det(\hat\Theta_{\mathrm{G}}) =1.2(1.1\cdot0.9-0.2^2)-0.3^2\cdot0.9 =1.2(0.99-0.04)-0.081 =1.059. $$ The determinant is positive, and the leading principal minors are $1.2$ and $1.2(1.1)-0.3^2=1.23$, so the symmetric matrix is positive definite. Its trace term is $$ \operatorname{tr}(H\hat\Theta_{\mathrm{G}}) =1.2+1.1+0.9+2\{0.4(-0.3)+0.1(0)+0.2(-0.2)\} =3.2-0.32=2.88. $$

Worked Example: Scoring the Graphical-Lasso Fit

Using the determinant and trace just calculated, $$ \bar\ell(\hat\Theta_{\mathrm{G}};H) =\frac12\{\log(1.059)-2.88\} \approx\frac12(0.05733-2.88) \approx-1.41134. $$ The logarithm is $\log(1.059)\approx0.05733$, and the reported score is rounded after using the unrounded logarithm. The constant omitted from every score is the same, so it does not change rankings.

For comparison, suppose the nodewise directed coefficients and residual variances have been converted to symmetric precision estimates using a prespecified averaging convention. The resulting union-rule estimate and intersection-rule estimate for this run are, respectively, $$ \hat\Theta_{\mathrm{U}}= \begin{pmatrix}1.10&-0.25&-0.15\\-0.25&1.00&-0.08\\-0.15&-0.08&0.95\end{pmatrix}, \qquad \hat\Theta_{\mathrm{I}}= \begin{pmatrix}1.10&-0.25&-0.15\\-0.25&1.00&0\\-0.15&0&0.95\end{pmatrix}. $$

For the union estimate, direct expansion gives $\det(\hat\Theta_{\mathrm{U}})=0.950085$. Its trace term is $3.05+2\{0.4(-0.25)+0.1(-0.15)+0.2(-0.08)\}=2.788$. Hence its score is $$ \frac12\{\log(0.950085)-2.788\}\approx\frac12(-0.05120-2.788)=-1.41960. $$

For the intersection estimate, $\det(\hat\Theta_{\mathrm{I}})=1.10(0.95)-0.25^2(0.95)-0.15^2(1.00)=0.963125$. Its trace term is $3.05+2\{0.4(-0.25)+0.1(-0.15)\}=2.82$. Thus $$ \frac12\{\log(0.963125)-2.82\}\approx\frac12(-0.0375721-2.82)\approx-1.42879. $$

These two nodewise-derived matrices are positive definite: their leading principal minors are positive, and their determinants, $0.950085$ and $0.963125$, are positive. On this held-out sample, the graphical-lasso estimate has the largest score among these three displayed estimates ($-1.41134$, compared with $-1.41960$ for the union estimate and $-1.42879$ for the intersection estimate). That is a result for this run, not a general ranking of the procedures.

What the Comparison Says—and Does Not Say

The example separates three decisions that are often blurred together. Cross-validation selected a penalty within each method. The nodewise symmetrization rule then changed its graph: the union retained one discordant edge that the intersection omitted. Finally, the held-out likelihood compared positive-definite precision estimates on a common scale. A favorable likelihood score does not by itself establish that the selected edges are correct; predictive fit and graph recovery are distinct goals.

A practical report should state the tuning grid and criterion, the nodewise symmetrization rule, and whether the precision matrix used for likelihood evaluation is positive definite. If symmetrizing produces a matrix that is not positive definite, the Gaussian likelihood formula cannot be applied to it as a valid precision matrix. Any repair, such as a specified positive-definite projection, changes the estimator and must be reported rather than silently assumed.

Common pitfall. It is tempting to treat an edge selected in one nodewise regression as automatically symmetric, or to compare raw penalty values across the two procedures. The first skips a modeling decision: the union and intersection rules can produce different graphs. The second ignores that the objectives penalize different quantities. State the symmetrization rule explicitly and compare tuned fits using a shared evaluation criterion.

Key takeaway: Tune each method using training data, define how directed nodewise selections become undirected edges, and evaluate the resulting positive-definite models on held-out data. Report edge selection and predictive likelihood as separate pieces of evidence.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

Why should graphical-lasso and nodewise-lasso penalty values not be compared directly?
For the directed selections in the example, which edge is selected by the union rule but not the intersection rule?
Show in one sentence why the intersection edge set must be a subset of the union edge set.
What condition on $\Theta$ is needed to interpret the held-out score as a Gaussian precision-model likelihood?
Why is choosing the penalty using the final held-out sample liable to overstate predictive performance?