Tutorials › Exam PA - Predictive Analytics › Why Ordinary P-Values Fail After Lasso Selection

Home  /  Tutorials  /  Exam PA / Lasso Selection and Elastic Net

Exam PA · Lasso Selection and Elastic Net · Tutorial 454 of 966

Why Ordinary P-Values Fail After Lasso Selection

Learn why post-lasso inference must account for the selection process, and how selection changes familiar p-values and confidence intervals.

Advanced 12 min read

What You'll Learn

  • Explain why a predictor selected using the outcome is not equivalent to a predictor specified in advance.
  • Derive a conditional p-value for a simple Gaussian screening rule.
  • Distinguish lasso shrinkage from the additional effect of selection on inference.
  • Calculate the effect of selecting the largest result from multiple candidates.
  • Identify what ordinary post-selection confidence intervals do and do not guarantee.
  • Recognize when sample splitting or selection-aware inference may be appropriate.

Why Selection Changes Inference

A p-value or confidence interval is calculated under assumptions about how the model and coefficient being examined were chosen. If the predictor was specified before looking at the outcome, standard inference can often be justified under a suitable model. If the same outcome data are used to search for predictors and then to report inference for the chosen predictors, that search is part of the statistical procedure. Ignoring it generally makes ordinary p-values too small and ordinary confidence intervals too narrow.

This matters after lasso because the fitted set of nonzero coefficients is data-dependent. The lasso is designed to produce a useful, often sparse predictive model; it does not automatically make conventional p-values or confidence intervals for its selected coefficients valid. Selection may also involve trying several penalty values or other modeling decisions, which can add further data dependence.

Definition: Post-selection inference is inference about model parameters after the data have been used to select a model, predictors, or other analysis choices. Ordinary post-selection inference applies standard p-value or confidence-interval calculations as if the selected model had been fixed in advance. It is generally unreliable unless the procedure justifies treating selection as independent of the inference data or explicitly accounts for selection.

A p-value is a tail probability calculated under a specified null hypothesis and a specified sampling procedure. It is not the probability that the null hypothesis is true. In particular, an ordinary p-value does not automatically account for the fact that a predictor was selected because its observed association looked unusually large.

A Simple Gaussian Selection Model

To see the issue precisely, suppose a candidate coefficient is estimated from data and standardized by its known standard error. Under a null coefficient and the usual normal sampling assumption, write the standardized estimate as \(Z\sim N(0,1)\). If this candidate was specified in advance, its two-sided p-value for an observed value \(z\) is \(2P(Z\geq |z|)\).

Now suppose the candidate is reported only when its standardized estimate passes a screening threshold \(c\), so selection occurs when \(|Z|>c\). Among reported candidates, values near zero are impossible: they could not have passed the screen. The sampling distribution after selection is therefore not the original standard normal distribution. It is the distribution of \(Z\) conditional on \(|Z|>c\).

Key result: Under the null \(Z\sim N(0,1)\), if the selection rule is \(|Z|>c\) and the observed value satisfies \(|z_{\mathrm{obs}}|>c\), the two-sided p-value conditional on selection is $$ p_{\mathrm{conditional}} = \frac{P(|Z|\geq |z_{\mathrm{obs}}|)}{P(|Z|>c)} = \frac{2[1-\Phi(|z_{\mathrm{obs}}|)]}{2[1-\Phi(c)]}. $$ Here \(\Phi\) is the standard normal cumulative distribution function. This formula applies to this particular one-statistic threshold rule under the stated null; it is not a universal correction for every lasso analysis.

The numerator is the probability, under the null, of observing a value at least as extreme as the one reported. Because that value already passed the screen, the same event is also a subset of the selection event. Conditional probability divides the probability of the extreme result by the probability of being selected at all.

Worked Example: A Coefficient Reported After Screening

Worked Example: A Coefficient Reported After Screening

A hypothetical analyst screens one standardized coefficient and retains it only if \(|Z|>1.5\). Assume the null model is correct, \(Z\sim N(0,1)\), and the observed value is \(z_{\mathrm{obs}}=2.20\). Compare the ordinary two-sided p-value with the p-value conditional on passing the screen.

Method and assumptions. Use the standard normal null distribution and the stated fixed screening threshold. The calculation treats the selected statistic as the reported result; it does not claim that this threshold formula applies to a general correlated-predictor lasso fit.

Step 1: Calculate the ordinary p-value. For a coefficient specified in advance, the two-sided p-value is \(2[1-\Phi(2.20)]\). Using the normal tail without first rounding the cumulative probability,

$$ p_{\mathrm{ordinary}} =2[1-\Phi(2.20)] \approx 0.0278069. $$

Step 2: Calculate the probability of selection under the null. The screen retains a value when its absolute value exceeds 1.5:

$$ P(|Z|>1.5) =2[1-\Phi(1.5)] \approx 2(0.0668072) =0.1336144. $$

Step 3: Condition on selection. Since \(2.20>1.5\), a value at least as extreme as 2.20 necessarily passes the screen. Thus the conditional p-value is the ordinary tail probability divided by the probability of passing the screen:

$$ p_{\mathrm{conditional}} = \frac{0.0278069}{0.1336144} \approx 0.2081. $$

The ordinary calculation gives about 0.0278, while the calculation that accounts for this screen gives about 0.2081. Common mistake avoided: using the ordinary p-value as if the coefficient had been chosen in advance. The two values answer different questions because the second accounts for the selection rule.

Why Lasso Selection Creates the Problem

In an orthogonal Gaussian design, the lasso makes the selection mechanism especially clear. Assume the predictors are orthogonal and standardized, the error terms are independent normal with known standard deviation, and consider one coefficient. Let \(Z\) be its unpenalized estimate divided by its standard error. For a suitable standardized penalty threshold \(c>0\), the lasso estimate on the same standardized scale is the soft-thresholded value

$$ \widehat{\theta}_{\mathrm{lasso}} = \operatorname{sign}(Z)(|Z|-c)_+, \qquad (a)_+=\max(a,0). $$

This formula follows from minimizing the one-coordinate penalized least-squares objective: the quadratic loss has derivative proportional to the difference between the unpenalized estimate and the coefficient, while the absolute-value penalty has derivative \(+c\) for a positive coefficient and \(-c\) for a negative coefficient. Setting the combined derivative to zero gives the estimate \(Z-c\) when \(Z>c\), \(Z+c\) when \(Z<-c\), and zero when \(|Z|\leq c\). The selected event is therefore \(|Z|>c\).

Even in this simple case, two issues must be separated. First, the lasso shrinks a selected estimate toward zero: the estimate is \(Z-c\) for \(Z>c\), not \(Z\). Second, the estimate was selected by exceeding the threshold, so its distribution is conditional on selection. A conventional standard error calculation that ignores either issue does not automatically give a valid p-value or confidence interval.

For a general lasso fit, predictor correlations, the selected signs, the penalty choice, and other modeling decisions can all affect the selection event. The simple threshold calculation above illustrates the principle, but it is not a plug-in formula for arbitrary lasso coefficients. The inference method must correspond to the actual selection procedure and its assumptions.

Worked Example: Lasso Shrinkage and Conditional Evidence

Worked Example: Lasso Shrinkage and Conditional Evidence

In a hypothetical orthogonal Gaussian analysis, a standardized coefficient estimate has \(Z=2.20\), and the standardized lasso threshold is \(c=1.50\). Assume the null distribution is \(N(0,1)\), as in the preceding model. Find the standardized lasso estimate, then compare the ordinary and selection-conditional p-values for the unpenalized statistic.

Method and assumptions. Apply the soft-thresholding formula for the lasso under an orthogonal design, then use the conditional-tail calculation for the rule \(|Z|>1.50\). The standard error is one on the standardized scale.

Step 1: Calculate the lasso estimate. Since \(Z>c\), soft thresholding subtracts the threshold:

$$ \widehat{\theta}_{\mathrm{lasso}} = 2.20-1.50 = 0.70. $$

Step 2: Calculate the ordinary p-value for \(Z\). If the coefficient had been specified in advance, the unpenalized statistic would have the ordinary p-value \(2[1-\Phi(2.20)]\approx 0.0278069\).

Step 3: Account for the selection rule. Under the null, the probability of passing the threshold is \(2[1-\Phi(1.50)]\approx 0.1336144\). Therefore,

$$ p_{\mathrm{conditional}} = \frac{2[1-\Phi(2.20)]}{2[1-\Phi(1.50)]} = \frac{0.0278069}{0.1336144} \approx 0.2081. $$

The lasso estimate is \(0.70\) on the standardized scale, but the ordinary p-value of about 0.0278 is not calibrated for a result reported only after this selection rule. Common mistake avoided: treating the shrunken estimate as if it were an unpenalized estimate with an unchanged standard error, or treating the unpenalized statistic as though it had not been selected.

Selection Also Affects Confidence Intervals

A conventional confidence interval is designed to have a stated coverage probability under repeated sampling for a fixed target and procedure. When the target is chosen from the data, coverage for the selected target can differ from the nominal level. In particular, the selected estimate tends to be more extreme than an estimate chosen in advance. An interval centered at that estimate may consequently be less likely to include the true value.

The next example uses independent estimates to quantify this effect. Independence is a simplifying assumption: it permits an exact calculation and makes clear why a pointwise 95% interval does not automatically retain 95% coverage after selecting the most extreme estimate.

Worked Example: Coverage After Choosing the Largest Estimate

A hypothetical analysis examines two independent standardized estimates \(Z_1\) and \(Z_2\). Assume both true coefficients are zero and \(Z_1,Z_2\) are independent standard normal variables. The analyst selects the estimate with the largest absolute value and reports its ordinary 95% interval, centered at the selected estimate with standard error one. What is the repeated-sampling probability that this selected interval contains zero?

Method and assumptions. A two-sided normal 95% interval contains zero exactly when the absolute value of its standardized estimate is at most 1.96. Since the selected estimate is the one with largest absolute value, its interval contains zero only if both estimates satisfy that condition.

Step 1: Express the coverage event. Let \(M=\max(|Z_1|,|Z_2|)\). The selected interval covers zero precisely when \(M\leq 1.96\), or equivalently when both \(|Z_1|\leq1.96\) and \(|Z_2|\leq1.96\).

Step 2: Use independence. Each estimate has 95% probability of lying between \(-1.96\) and \(1.96\). Independence gives

$$ P(M\leq1.96) = P(|Z_1|\leq1.96)P(|Z_2|\leq1.96) = 0.95^2 = 0.9025. $$

Thus the selected interval covers zero in 90.25% of repeated samples, not 95%, under these assumptions. A conservative simultaneous approach would use 97.5% pointwise intervals for each coefficient: by the union bound, the probability that either misses its true value is at most \(0.025+0.025=0.05\). The corresponding normal critical value is about 2.2414. Common mistake avoided: assuming that an interval with 95% pointwise coverage still has 95% coverage after the estimate to report has been chosen for being most extreme.

What Can Be Done Instead?

The appropriate response depends on the objective, the selection procedure, and the data available. A predictive model may be useful even when ordinary coefficient p-values are not justified. If the goal is inference, the report should say how selection was handled rather than presenting conventional output as if the model had been fixed in advance.

  • Use separate data for selection and estimation. Select predictors in one sample and estimate or test them in an independent sample. Conditional on the selection sample, the independent inference sample has not been used to choose those predictors. This approach relies on an appropriate model for the inference sample and may lose precision because fewer observations are available at each stage.
  • Use a method that accounts for the selection event. Selection-aware inference conditions on the event that produced the selected model. Its validity depends on correctly specifying the model, selection procedure, and sampling assumptions.
  • Report predictive performance separately from coefficient inference. A selected predictor can help prediction without supporting a conventional claim about its coefficient. Predictive usefulness and inferential evidence are distinct questions.
  • State the scope of the conclusion. Explain which predictors were searched, how they were selected, and whether reported uncertainty accounts for that search. Avoid interpreting a selected association as causal evidence merely because its ordinary p-value is small.

These approaches do not make selection disappear. They make the role of selection explicit and align the inferential claim with the actual analysis. The central principle is simple: an inferential calculation must reflect not only how a coefficient was estimated, but also how it came to be the coefficient reported.

Key takeaway: After lasso selection, ordinary p-values and confidence intervals are generally unreliable when they treat the selected model as fixed in advance. Selection changes the sampling distribution; lasso additionally shrinks estimates. Valid inference requires a method justified for the selection procedure, or independent data for inference.

Check Your Understanding

For each question, assume the stated normality, independence, and selection conditions hold. Answers use probabilities rounded to four decimal places unless stated otherwise.

  1. Two independent null standardized estimates are examined, and the one with the largest absolute value is reported. If its observed absolute value is 2.00, calculate the ordinary two-sided p-value and the probability that at least one of the two estimates is at least this extreme. (Answer: ordinary p-value \(=0.0455\); selection-adjusted probability \(=1-[1-0.0455003]^2\approx0.0889\).)
  2. Three independent null standard normal statistics are examined, and the largest absolute value is 2.40. Calculate the ordinary two-sided p-value and the probability that at least one statistic is at least this extreme. (Answer: ordinary p-value \(=2[1-\Phi(2.40)]\approx0.0164\); adjusted probability \(=1-(1-0.0163951)^3\approx0.0484\).)
  3. A single null statistic is reported only when \(|Z|>1.00\). It is observed to be \(Z=1.80\). Calculate its two-sided p-value conditional on selection. (Answer: \([1-\Phi(1.80)]/[1-\Phi(1.00)]\approx0.2264\).)
  4. Under the orthogonal Gaussian lasso model, let the null standardized statistic be \(Z\sim N(0,1)\), with threshold \(c=1.00\). Given selection, calculate the expected absolute value of the lasso estimate \((|Z|-1)_+\). Use \(E[|Z|\mathbf{1}(|Z|>1)]=2\phi(1)\) and \(\phi(1)\approx0.2420\). (Answer: \(\phi(1)/[1-\Phi(1)]-1\approx0.5251\).)
  5. A predictor set is selected using one sample, and a separate independent sample is used to construct a 95% confidence interval for a coefficient in that fixed selected set. Under the assumptions supporting the usual interval in the inference sample, what is its conditional coverage given the selection sample? (Answer: 95%, provided the independent inference sample and model assumptions are valid.)

Previous

Next

Back to Tutorials
Clinical Biostats