Why Compare the Three Penalties on the Same Genes?
Gene-expression predictors often move together. If two genes are highly correlated, their individual contributions to a fitted response may be hard to distinguish even when their combined predictive contribution is clear. Ridge, lasso, and elastic net respond differently to this situation: ridge shrinks coefficients without setting them exactly to zero; lasso can set coefficients to zero and select features; elastic net combines these penalties and can encourage correlated genes to enter together.
The calculations below isolate those differences using a small hypothetical example. The same algebra applies when the number of genes exceeds the number of samples, although this three-gene example keeps the matrix calculations visible. The goal is to compare coefficient patterns and fitted values under specified penalties, not to claim that one fit will predict best on new samples.
Model, Scaling, and Notation
Assume a continuous response \(y\) and \(p\) gene-expression predictors arranged in the matrix \(X\). The response and each predictor column are centered, so an intercept can be omitted from the calculations; in an uncentered model, the intercept is fitted but not penalized. Predictors are scaled to have mean square one, so their cross-product matrix \(G=X^\mathsf{T}X/n\) has ones on its diagonal. This scaling matters because penalties act on coefficient magnitudes. The response need not be standardized.
We use the following objective conventions. The elastic-net mixing parameter is \(0\leq\alpha\leq1\); \(\alpha=1\) gives lasso and \(\alpha=0\) gives ridge with this scaling. All three procedures use the residual sum of squares divided by \(2n\), so their objectives can be expressed through \(G\), \(c\), and \(q\).
For any coefficient vector, its training mean squared error is \(q-2c^\mathsf{T}\beta+\beta^\mathsf{T}G\beta\). Ridge satisfies \((G+\lambda I)\hat\beta=c\). For lasso, a zero coefficient is allowed when its residual cross-product is within the threshold \(\lambda\); a nonzero coefficient also pays the absolute-value penalty. These facts let us calculate the fits without constructing every row of \(X\).
Worked Example: A Strongly Correlated Gene Pair
Worked Example: Three Penalties on the Same Hypothetical Data
Suppose three centered, scaled gene predictors have cross-product matrix and response cross-products
These are hypothetical sufficient statistics. Genes 1 and 2 have correlation \(0.9\); gene 3 is uncorrelated with each of them in this design. The response-predictor cross-products are \(1,1,0.2\), not Pearson correlations. For example, the Pearson correlation of gene 1 with the response is \(1/\sqrt{1.09264}\approx0.9567\). Take \(\lambda=0.2\) for all three fits, and \(\alpha=0.5\) for elastic net.
Ridge. The first two coefficients are equal by symmetry. Their equations are \(1.2\beta_1+0.9\beta_2=1\) and \(0.9\beta_1+1.2\beta_2=1\), while \(1.2\beta_3=0.2\). Therefore
Lasso. For the first two genes, the symmetric positive solution has \(1.9b=1-0.2\), so \(b=0.8/1.9=0.42105\). This is the correct symmetric lasso calculation: the two-coordinate equation uses the sum of both coefficients, and with equal coefficients it is \(1.9b=1-0.2\), giving \(b=0.42105\). Check this against the objective's gradient: each coordinate has \(G\beta=1.9b\), so the equation is indeed \(1.9b=0.8\). Thus the correct lasso coefficients for genes 1 and 2 are \(0.42105\) each, not \(0.4\). Gene 3 has cross-product exactly \(\lambda\), so a zero coefficient satisfies the lasso threshold condition. The fit is
Elastic net. With \(\alpha=0.5\), the L1 and L2 parts each have coefficient \(0.1\). For the first two genes, \((1.9+0.1)b=1-0.1\), giving \(b=0.45\). For gene 3, \((1+0.1)\beta_3=0.2-0.1\), giving \(\beta_3=0.09091\). Hence
The resulting training mean squared errors use the same expression for each fit. Ridge gives \(1.09264-2(0.985714)+0.889456\approx0.01067\). Lasso gives \(1.09264-2(16/19)+3.8(8/19)^2\approx0.08211\). Elastic net gives \(1.09264-2(0.91818)+0.77776\approx0.03404\). These are in-sample errors, not estimates of error for new samples. In particular, the smaller ridge training error does not establish that ridge predicts better outside this hypothetical data set.
The lasso arithmetic deserves care: when two coefficients are both \(b\), each gene's fitted cross-product is \(b+0.9b=1.9b\), not \(2b\). Also, the squared term for elastic net is \(3.8(0.45)^2+(0.09091)^2\approx0.77776\). Keeping the matrix multiplication explicit prevents errors in the reported training errors.
Why Correlation Changes the Coefficient Pattern
The role of correlation is especially clear for genes 1 and 2. Let \(t=\beta_1+\beta_2\) denote their sum and \(d=\beta_1-\beta_2\) their difference. Their contribution to the quadratic predictor term is
When the genes are highly correlated, the data-fit term penalizes a difference between their coefficients relatively weakly: the coefficient on \(d^2\) is only \(0.05\). Ridge adds an L2 penalty that discourages a large difference as well as a large sum, so it tends to distribute signal across correlated genes. For same-sign positive coefficients, the lasso penalty is \(|\beta_1|+|\beta_2|=t\): it does not itself distinguish how a fixed positive sum is divided. Small data perturbations can therefore change which correlated gene receives the nonzero coefficient. Elastic net adds a squared-coefficient penalty, which discourages unequal allocation and can encourage a correlated group to be represented together. This grouping tendency is not a guarantee that every correlated gene will have a nonzero coefficient.
These observations are consistent with the earlier material on lasso nonuniqueness under collinearity and the elastic-net grouping effect. Here the emphasis is on comparing the actual fits: similar fitted values can arise from substantially different coefficient patterns.
Worked Example: A Small Asymmetry Can Change Selection
Worked Example: Perturbing the Cross-Products
Keep \(G\) as above, but consider hypothetical response cross-products \(c=(1.10,0.98,0)\) and response mean square \(q=1.30\). These cross-products are feasible: the minimum possible response mean square for this \(c\) is \(c^\mathsf{T}G^{-1}c\approx1.21053\), which is less than \(1.30\). Again use \(\lambda=0.2\), with \(\alpha=0.5\) for elastic net.
For ridge, solving the first two equations \(1.2\beta_1+0.9\beta_2=1.10\) and \(0.9\beta_1+1.2\beta_2=0.98\) gives \((\hat\beta_1,\hat\beta_2)\approx(0.69524,0.29524)\); the third coefficient is zero. For lasso, a fit with only gene 1 active has \(\hat\beta_1=1.10-0.2=0.90\). At that fit, gene 2's residual cross-product is \(0.98-0.9(0.90)=0.17\), whose absolute value is below \(0.2\); thus gene 2 can remain zero. Elastic net solves
The estimates show three patterns: ridge retains both genes with unequal coefficients, lasso retains only the gene with the larger cross-product, and elastic net retains both but shrinks them. For a profile with standardized gene values \((1,1,0)\), the three centered fitted values are respectively \(0.99048\), \(0.90\), and \(0.94\). The fitted values are relatively close even though the lasso selects a different set of genes. That is a concrete reason not to treat a selected feature list as a complete description of a predictive fit.
Worked Example: Comparing Fitted Values and Selected Features
Worked Example: One Hypothetical New Expression Profile
Return to the first example and consider a hypothetical centered profile with standardized gene values \(x_*=(1,1,1)\). The fitted response is \(x_*^\mathsf{T}\hat\beta\), because the data were centered and the intercept is zero in this representation.
Ridge gives \(0.47619+0.47619+0.16667=1.11905\). Lasso gives \(0.42105+0.42105+0=0.84211\). Elastic net gives \(0.45+0.45+0.09091=0.99091\). These are fitted values in the response's units. The corresponding nonzero feature sets are all three genes for ridge, genes 1 and 2 for lasso, and all three genes for elastic net. Ridge generally does not induce sparsity, although an individual coefficient can be exactly zero for particular \(G\) and \(c\); lasso and elastic net can.
This is one profile, not a test-set evaluation. A single fitted value cannot establish which method has lower prediction error. Further, a coefficient is a conditional contribution within the fitted model, not evidence that the gene causes a change in the response. Highly correlated genes can make individual coefficients especially difficult to interpret biologically.
Assumptions, Diagnostics, and Limitations
The calculations assume the same centered data, gene scaling, response, and loss scaling for all methods. They also assume a continuous response and squared-error loss. The first example's matrix is positive definite, so the ridge solution is unique; in larger \(p>n\) settings, the predictor matrix can be rank deficient, but the positive ridge penalty still gives a unique ridge coefficient solution. Lasso coefficients need not be unique in every rank-deficient design, even when fitted values are unique.
Before interpreting a comparison, check that predictor scaling was applied consistently and that the reported \(G\), \(c\), and \(q\) match the data. Verify that \(G\) is symmetric with nonnegative quadratic forms and that the response cross-products are not being mislabeled as correlations. Check the KKT residual condition for coefficients reported as zero, particularly when a cross-product is close to \(\lambda\). A coefficient that is nearly zero may be sensitive to small changes in the data or numerical tolerance.
The shared value \(\lambda=0.2\) in these examples makes the arithmetic comparable, but it does not make the penalties equally tuned or equally complex. Nor does it imply that the best \(\lambda\) is the same for different methods. Practical comparisons should use a consistent scaling and a defensible assessment of prediction on data not used to fit the coefficients; the present examples illustrate mechanisms, not a tuning rule.
| Penalty | Typical pattern with correlated genes | Interpretive caution |
|---|---|---|
| Ridge | Shrinks coefficients, generally retaining all predictors | Nonzero coefficients do not by themselves identify a short gene list |
| Lasso | Can set coefficients exactly to zero; may select one gene from a correlated group | Which gene is selected can be sensitive to small cross-product differences |
| Elastic net | Combines sparsity with an L2 grouping tendency | Grouping is encouraged, not guaranteed; results depend on \(\lambda\) and \(\alpha\) |
Common mistake: Seeing two correlated genes with one lasso coefficient equal to zero and concluding that the selected gene is uniquely important, while the omitted gene is irrelevant. The correction is that lasso selection describes one penalized fit under a particular scaling, penalty, and data realization. With correlated predictors, a small change in the response cross-products can change the selected member while leaving predictions similar.
Check Your Understanding
Use the definitions and calculations above to answer these questions. Brief solutions follow each prompt.
- With centered predictors scaled to mean square one, \(q=1.44\), and \(c_j=0.72\), what is the Pearson correlation between predictor \(j\) and the response? Solution: It is \(0.72/\sqrt{1.44}=0.72/1.2=0.600\), not \(0.72\).
- Why can ridge have nonzero coefficients for genes that contribute little individually? Solution: Ridge shrinks coefficients through an L2 penalty but does not generally set them exactly to zero. It can distribute a shared signal across correlated predictors.
- In the second worked example, why is a zero lasso coefficient for gene 2 permissible when gene 1 has coefficient \(0.90\)? Solution: Gene 2's residual cross-product is \(0.98-0.9(0.90)=0.17\), and \(|0.17|\leq\lambda=0.2\), satisfying the zero-coefficient threshold condition.
- For the first example's elastic-net fit, verify the quadratic term \(\hat\beta^\mathsf{T}G\hat\beta\). Solution: The correlated pair contributes \(3.8(0.45)^2=0.7695\), and gene 3 contributes \((0.09091)^2\approx0.008264\). Their sum is approximately \(0.77776\).
- Does a smaller training mean squared error in the first example establish better prediction for future samples? Solution: No. Training error is calculated on the data used to fit the coefficients; assessing prediction on new samples requires an appropriate out-of-sample evaluation.