The Winner's Curse In GLM Selection

Why the Winning GLM Can Look Too Good

A candidate GLM can appear strongest because it captures a real pattern, because its estimate is noisy in a favorable direction, or both. Searching over many candidates gives that noise many chances to produce a low loss, a high score, or a small p-value. The model selected as the “winner” therefore needs to be distinguished from a model whose apparent advantage is supported by independent evidence.

The same mechanism affects coefficients. If many candidate effects are estimated and the largest-looking one is reported, its estimate tends to include unusually favorable sampling error. This is the winner’s curse: selection makes the winner’s observed performance or effect size more optimistic than its underlying value. The amount of distortion depends on the candidate set, the variability of the estimates, and their dependence.

Selection Optimism for a Finite Candidate Set

Definition: Consider a fixed finite set of candidate models \(M_1,\ldots,M_K\). Let \(R_j\) be the target risk of candidate \(M_j\), where smaller is better, and let \(S_j\) be its estimated risk. Select \(\hat j=\operatorname{argmin}_{1\leq j\leq K} S_j\), using a fixed rule to break ties. The selection optimism is \(E[R_{\hat j}-S_{\hat j}]\). A positive value means the selected model’s reported risk is, on average, below its target risk.

The following result isolates the effect of selection. It does not claim that every practical score is unbiased: it assumes \(E[S_j]=R_j\) for each candidate. For example, conditional on data used to fit a fixed set of candidate models, an independent validation sample can provide unbiased average loss estimates for their conditional risks. Scores computed on the same data used to fit or construct the candidates may have additional bias.

Theorem: If the candidate set is finite and each score has finite expectation with \(E[S_j]=R_j\), then \(E[R_{\hat j}-S_{\hat j}]\geq 0\). Independence among the scores is not required.

Bound the selected target risk.
For every outcome, \(R_{\hat j}\geq \min_j R_j\), so \(E[R_{\hat j}]\geq \min_j R_j\).

Bound the selected estimated risk.
For every outcome, \(\min_j S_j\leq S_k\) for every fixed \(k\). Taking expectations gives \(E[\min_j S_j]\leq E[S_k]=R_k\). This holds for every \(k\), hence \(E[S_{\hat j}]=E[\min_j S_j]\leq\min_j R_j\).

Compare the bounds.
Subtracting gives \(E[R_{\hat j}]-E[S_{\hat j}]\geq \min_j R_j-\min_j R_j=0\), as claimed. □

This result shows why an estimated score for the selected model is not automatically a fair estimate of its performance. It does not say that selection must create strict optimism: if all scores are identical without noise, the difference can be zero. Nor does it say that the selected model’s target risk must exceed the risk of every other candidate. The inequality concerns the gap between its target risk and its selected estimated risk, in expectation.

Worked Example: Selecting the Lower of Two GLM Risk Estimates

Suppose two candidate Poisson GLMs have the same target risk, \(R_1=R_2=10\). For illustration, assume their validation risk estimates are \(S_j=10+\varepsilon_j\), where \(\varepsilon_1,\varepsilon_2\) are independent \(N(0,4)\) variables. Thus each estimate is unbiased and has standard deviation 2. The selected estimate is the smaller of the two.

Use \(\min(a,b)=(a+b-|a-b|)/2\). Since the errors have mean zero,

E[\min(S_1,S_2)] =10-\frac{1}{2}E[|\varepsilon_1-\varepsilon_2|].

Independence and the stated variances give \(\varepsilon_1-\varepsilon_2\sim N(0,8)\). If \(Z\sim N(0,1)\), then \(E[|Z|]=\sqrt{2/\pi}\), so \(E[|N(0,8)|]=\sqrt{8}\sqrt{2/\pi}=4/\sqrt{\pi}\). Therefore,

E[\min(S_1,S_2)]=10-\frac{2}{\sqrt{\pi}}\approx 8.872.

Whichever candidate is selected, its target risk is 10. The expected reported risk is about 8.872, giving expected optimism \(2/\sqrt{\pi}\approx1.128\). This calculation illustrates the selection effect under the specified score model; it is not a universal correction to apply to validation scores.

Smallest P-Values and Multiple Opportunities

Searching for a small p-value is another form of choosing an extreme score. The assumptions matter. A valid p-value under a null hypothesis is super-uniform: for every \(t\in[0,1]\), \(P(p\leq t)\leq t\). Exact uniformity is stronger. A p-value calculated from an exact continuous null distribution is Uniform\((0,1)\) under that null; validity alone does not imply uniformity. For instance, a procedure that always reports 1 is valid but is not exactly uniform.

Theorem: If \(p_1,\ldots,p_K\) are independent Uniform\((0,1)\) random variables under their respective null hypotheses, then for \(p_{\min}=\min_j p_j\), \(P(p_{\min}\leq t)=1-(1-t)^K\) for \(0\leq t\leq1\), and \(E[p_{\min}]=1/(K+1)\).

Proof. The event \(p_{\min}>t\) means that every p-value exceeds \(t\). By independence and uniformity, \(P(p_j>t)=1-t\), so \(P(p_{\min}>t)=(1-t)^K\). Taking the complement proves the distribution formula. For a nonnegative random variable bounded by 1, \(E[p_{\min}]=\int_0^1P(p_{\min}>t)\,dt\). Substituting the survival probability just derived gives

E[p_{\min}]=\int_0^1(1-t)^K\,dt=\frac{1}{K+1}.

The last equality follows by integrating: \(\int_0^1(1-t)^K\,dt=[-(1-t)^{K+1}/(K+1)]_0^1=1/(K+1)\). □

When p-values are merely valid and possibly dependent, the exact formula need not hold. The union bound still gives \(P(\min_j p_j\leq t)\leq\sum_jP(p_j\leq t)\leq Kt\). This bound may be conservative; independence and exact uniformity are what justify the formula above.

Worked Example: Ten Independent Exact Tests

Imagine testing ten prespecified null hypotheses, with independent exact p-values that are Uniform\((0,1)\) when all ten nulls hold. The probability that at least one p-value is at most 0.05 is

P(p_{\min}\leq0.05)=1-(1-0.05)^{10}=1-0.95^{10}\approx0.4013.

Thus a minimum p-value at or below 0.05 occurs about 40.1% of the time under these assumptions, not 5% of the time. The expected minimum is \(1/(10+1)=1/11\approx0.0909\). These calculations apply to independent exact p-values; candidate GLM tests often share data and predictors, so independence cannot be presumed. Dependence changes the exact probability, although the union-bound guarantee for valid p-values remains available.

Selected Coefficients Are Also Extremes

The same reasoning applies to effect estimates. A useful simplified model is \(\hat\beta_j=\beta_j+\varepsilon_j\), where the candidate-specific errors have mean zero. GLM maximum-likelihood estimates can be approximately normal under standard regularity conditions, such as a correctly specified model, an interior true parameter, and sufficient information. The simple model below makes the selection mechanism exact rather than relying on that approximation.

Worked Example: Choosing the Larger of Two Estimated Effects

Suppose two candidate logistic-regression coefficients both have true value zero. As a stylized approximation to their estimation errors, let \(\hat\beta_1\) and \(\hat\beta_2\) be independent \(N(0,0.25^2)\) variables. Select the candidate with the larger estimated coefficient, \(\hat\beta_{\hat j}=\max(\hat\beta_1,\hat\beta_2)\). Write each estimate as \(0.25Z_j\), where \(Z_1,Z_2\) are independent standard normal variables.

The identity \(\max(a,b)=(a+b+|a-b|)/2\) gives

E[\hat\beta_{\hat j}] =\frac{0.25}{2}E[Z_1+Z_2+|Z_1-Z_2|] =\frac{0.25}{2}E[|Z_1-Z_2|].

Here \(Z_1-Z_2\sim N(0,2)\), so \(E[|Z_1-Z_2|]=\sqrt{2}\sqrt{2/\pi}=2/\sqrt{\pi}\). Hence

E[\hat\beta_{\hat j}]=\frac{0.25}{\sqrt{\pi}}\approx0.1410.

The selected estimate is positive on average even though both true coefficients are zero. In this example, the selected candidate’s true coefficient is always zero, so the entire expected selected estimate is selection bias. The calculation concerns choosing the larger signed estimate; selecting by the smallest two-sided p-value instead favors a large absolute estimate and has a related, but not identical, distribution.

In a real GLM search, candidate coefficients and scores are usually dependent: models may share observations, covariates, or nested terms. Dependence affects the size of the winner’s curse, not the basic warning that selection on noisy estimates can exaggerate the selected result. The simple normal example quantifies one case; it should not be treated as a universal bias correction.

Practical Interpretation and a Common Mistake

The winner’s curse matters whenever a selected model is evaluated using the same information that helped select it. A low training loss, a leading cross-validation score, or a striking coefficient can reflect both signal and favorable noise. The selection-optimism theorem requires unbiased candidate scores for its target risks; practical procedures may not meet that assumption exactly. The broader lesson is to distinguish the selection criterion from an independent assessment of the selected model.

A common mistake is to say, “Each null p-value is valid at 0.05, so the selected p-value is also valid at 0.05.” This confuses a prespecified test with a search over tests. For \(K\) independent exact null p-values, the chance that the minimum is at most 0.05 is \(1-0.95^K\), not 0.05. With only validity and arbitrary dependence, one can conclude the upper bound \(K(0.05)\), capped at 1, from the union bound—not the independent-uniform formula. Likewise, a selected coefficient’s ordinary estimate does not become unbiased merely because each candidate estimate was unbiased before selection.

Key takeaway: Selection favors favorable noise. The expected reported risk of the winner can be below its target risk, and an effect chosen for looking unusually large can be exaggerated. Quantify these effects only under stated assumptions, and do not interpret the selected score or p-value as though its candidate had been fixed in advance.

Check Your Understanding

Use the assumptions in each question explicitly; do not assume independence or exact uniformity unless stated.

State the conditions in the selection-optimism theorem and explain why independence of the candidate risk estimates is not needed.
For two independent exact null p-values, derive the probability that their minimum is at most \(t\).
Why does the fact that a p-value is valid not, by itself, imply that it is Uniform\((0,1)\)?
In the two-coefficient example, what feature of the selection rule makes the selected estimate positive on average despite zero true effects?
If candidate p-values are dependent but each is valid, what upper bound can the union bound provide for the probability that their minimum is at most \(t\)?