Selecting Sparsity by Held-Out Association

Why Tune Sparsity With Held-Out Association?

Sparse CCA and sparse PLS seek cross-block relationships while restricting the number of variables that contribute to their scores. The penalty controls that restriction: a stronger penalty will generally produce fewer nonzero coefficients, but it can also remove variables needed to capture a reproducible relationship. Choosing a penalty only by the association measured on the data used to fit the model is risky, because the optimization has already selected coefficients to make that association large.

Held-out tuning addresses this problem by separating model fitting from evaluation. For each candidate penalty, fit the sparse method using training samples, then apply the fitted score directions to matched validation samples that were not used in fitting. Compare the resulting cross-block score association across candidate penalties. The aim is not to prove that the selected model has a true association, but to choose a level of sparsity whose score relationship generalizes better to new matched observations.

Let \(X\) and \(Y\) be two measured blocks for the same \(n\) individuals. In a training set, a fitted sparse model produces coefficient vectors \(a\) and \(b\). For held-out individual \(i\), the corresponding scores are \(u_i=x_i^\mathsf{T}a\) and \(v_i=y_i^\mathsf{T}b\), using the fitted preprocessing and coefficients. The validation association is

where \(m\) is the number of held-out matched individuals, and \(\bar{u}\) and \(\bar{v}\) are the means of their validation scores. These means are used to calculate the validation correlation, not to refit or recenter the training model. A validation score uses the same feature centering, scaling, and coefficient directions learned from the training data.

A Valid Evaluation Protocol

Before tuning, specify the candidate penalty values, the held-out association criterion, and how ties or near-ties will be resolved. In each fold, split matched individuals into training and validation sets. Fit every data-dependent operation on the training set alone: this includes imputation, centering, scaling, feature filtering, and estimating sparse score directions. Apply those learned transformations to validation observations, calculate validation scores, and then calculate their association. A validation observation must not help determine which variables are retained or what coefficient directions are used.

For an association criterion, the score orientation must also be handled consistently. Sparse CCA directions can have arbitrary signs. Orient the fitted directions using the training data, for example so that the training cross-block association is positive, and keep those orientations fixed in validation. Do not flip a validation score direction after seeing its validation association just to turn a negative value positive: that would use the held-out result to improve the evaluation.

When folds have equal sizes, a simple summary for penalty \(\lambda\) is the mean of the fold-specific validation correlations. With unequal fold sizes, a prespecified size-weighted summary or a Fisher-transformed summary can be appropriate. The summary rule should be used for all candidates. Cross-validation folds are not independent replicates, especially when training sets overlap, so variation across folds is a useful measure of tuning instability, not automatically a formal confidence interval for future performance.

Calculating a Held-Out Score Association

Comparing Fit With Compactness

A tuning curve plots a validation association summary against penalty strength or, often more interpretably, against the number of nonzero coefficients. The candidate with the largest mean held-out association favors fit. A compactness-oriented choice may accept a small decrease in association in exchange for a smaller support. One common compromise is the one-standard-error rule: find the candidate with the highest mean validation score, subtract its estimated standard error from that maximum, and choose the sparsest candidate whose mean score is at least that threshold. This is a selection convention, not a theorem that the chosen model is optimal.

The standard error used in this rule is often calculated from the fold-level scores as their sample standard deviation divided by the square root of the number of folds. Because folds share training data, this quantity is best treated as a pragmatic tuning threshold, not as a fully calibrated uncertainty interval. The rule is most straightforward when all candidates use the same folds and the same scoring convention.

The rule should not obscure the tuning curve itself. A broad plateau suggests that several penalties produce similar held-out associations, while a sharp isolated maximum may signal instability or sensitivity to a few observations. Support size is a useful measure of compactness, but equal support sizes do not imply equal models or equal stability. The subsequent interpretation of selected variables requires separate care.

What Association Is Being Optimized?

For sparse CCA, the natural target is correlation between the two block scores. Sparse PLS is based on cross-block covariance, so a covariance-based criterion may appear more aligned with its fitting objective. However, covariance depends on the scale of the scores and the blocks. If score scaling differs between folds or candidates, covariance comparisons can reflect scale as well as association. A correlation criterion is scale-invariant and can make candidates easier to compare when the practical goal is the strength of standardized score association.

The criterion must match the scientific goal and be fixed before comparing candidates. Do not select whichever metric gives the preferred answer after inspecting the tuning results. If covariance is used, state the score scaling and preserve it consistently. If correlation is used, remember that it measures linear association, not the amount of cross-block variation explained or a causal effect.

Diagnostics, Failure Modes, and Alternatives

A tuning result is only as credible as its data separation and validation design. Inspect the fold-level scores, not just their mean. Large fold-to-fold variation, sign reversals, or a maximum that changes markedly when folds are reassigned indicate that the selected penalty is uncertain. Check that each validation fold contains enough matched individuals for a meaningful correlation and that neither score vector is constant; correlation is undefined if either validation score has zero variance.

Common failures include fitting preprocessing once on the full data before cross-validation, choosing features using all observations, and repeatedly checking a final test set while adjusting penalties. Each allows information from held-out observations to influence the model or the selection. Another failure is to break the match between rows of \(X\) and \(Y\), which changes the paired association being evaluated. If samples have repeated measures, family structure, or other dependence, folds should keep dependent observations together; random row-wise splitting can otherwise place closely related observations in both training and validation sets.

The simplest alternative is to maximize the mean validation association. It is transparent and useful when predictive association is the primary objective, but it can select a larger, less stable support for a very small gain. The one-standard-error rule favors compactness when candidates perform similarly, but its fold-based standard error is only a heuristic. A separate test set gives a clean final evaluation if it is used once after all tuning decisions. Nested cross-validation repeats the tuning inside each outer training set and evaluates the selected procedure on the corresponding outer fold; it is more computationally demanding, but it assesses the selection process rather than treating a tuned validation maximum as an unbiased performance estimate.