Shrinkage, Winner's Curse, And Selected Marker Effects

Why Selected Marker Effects Look Too Large

A marker gene is often reported because its observed contrast between clusters is large. But an observed contrast combines the gene’s underlying effect with sampling noise. When selection favors extreme observations, it also favors observations whose noise happened to point in the selected direction. Consequently, the reported effects tend to overstate the underlying effects. This is the winner’s curse.

This is an estimation problem, distinct from testing whether a marker passes a significance or false-discovery-rate threshold. A procedure can control a specified error rate while the estimated effects of selected genes remain exaggerated. Conversely, shrinking estimates can reduce average estimation error without making a test or interval valid after selection.

For a cluster marker, \(\theta_g\) might represent the population difference in mean expression between a target cluster and a comparison group, measured on a specified analysis scale. A useful approximation for discussing estimation is \(\hat{\theta}_g\mid\theta_g\sim N(\theta_g,s_g^2)\), where \(s_g\) is the standard error. This is a working model, not an automatic description of raw single-cell counts: the contrast and its uncertainty must come from an analysis appropriate to the data and study design.

How Selection Creates the Exaggeration

Suppose several genes have the same true effect and independent estimation errors. If the gene with the largest observed positive contrast is reported, then its error is the largest of several errors, not a typical error. Even when every true effect is zero, the largest observed estimate will usually be positive.

The same mechanism applies when selection is based on a threshold rather than a rank. Among genes whose estimates exceed a positive threshold, estimates near the threshold are more common than arbitrarily large estimates, but their errors must on average be positive if their true effects are fixed at zero. The thresholded sample therefore has a positive mean estimate even under the null.

A Basic Shrinkage Estimator

Shrinkage replaces a noisy estimate with a value pulled toward a reference, often zero for a contrast. A common model makes that rule explicit: assume \(\hat{\theta}_g\mid\theta_g\sim N(\theta_g,s_g^2)\) and, across genes, \(\theta_g\sim N(0,\tau^2)\). For this simple normal-means model, suppose \(s_g\) and \(\tau\) are known. The prior says effects are centered at zero and have variance \(\tau^2\); it is a modeling assumption about effects across genes, not a fact guaranteed by the data.

The factor \(\lambda_g\) is smaller when the estimate is less precise relative to the assumed spread of true effects. If \(s_g^2\) is large relative to \(\tau^2\), the data provide comparatively little information about that gene’s effect and the estimate moves more toward zero. If \(s_g^2\) is small relative to \(\tau^2\), the estimate stays closer to the observed contrast.

The formula follows by multiplying the normal likelihood by the normal prior and completing the square. Ignoring factors that do not depend on \(\theta_g\), the posterior density is proportional to \[ \exp\left\{-\frac{1}{2}\left[ \frac{(\hat{\theta}_g-\theta_g)^2}{s_g^2} +\frac{\theta_g^2}{\tau^2}\right]\right\}. \] The coefficient of \(\theta_g^2\) inside the brackets is \(1/s_g^2+1/\tau^2\), so the posterior variance is its reciprocal. The coefficient of the linear term is \(\hat{\theta}_g/s_g^2\); multiplying it by that variance gives the posterior mean: \[ \frac{\hat{\theta}_g/s_g^2}{1/s_g^2+1/\tau^2} =\frac{\tau^2}{\tau^2+s_g^2}\hat{\theta}_g. \] This derivation establishes the formula only under the stated model.

What Shrinkage Can—and Cannot—Correct

Shrinkage can reduce the influence of noisy extremes. Under the normal-means model, the posterior mean is optimal for squared-error loss when the prior and likelihood are correct. More generally, a well-calibrated shrinkage rule can improve average estimation accuracy across a collection of genes. That is not the same as removing the selection mechanism. A selected estimate can remain exaggerated after shrinkage, especially if the shrinkage is mild, the selection is severe, or the prior is misspecified.

For a transparent illustration, let \(Y\sim N(0,1)\) for a gene whose true effect is zero, and report it only when \(Y>1.5\). Under this threshold rule, the conditional mean of the reported estimate is \(E[Y\mid Y>a]=\phi(a)/(1-\Phi(a))\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. This identity follows by integrating the normal density: \(\int_a^\infty y\phi(y)\,dy=\phi(a)\), and dividing by \(P(Y>a)=1-\Phi(a)\).

There is an important distinction between a model-based posterior statement and repeated-sampling bias at a fixed true effect. If the prior describes the gene effects well and the likelihood is correct, then posterior means have a conditional-expectation interpretation given the observed data. But estimates selected from a particular realized data set can still be misleading if the model omits relevant structure, uses poorly estimated hyperparameters, or treats adaptive clustering and marker selection as though they had not occurred. Shrinkage is an estimation tool, not a universal post-selection correction.

Assumptions, Diagnostics, and Alternatives

Before using the normal-means shrinkage formula, check that the contrast and standard error are meaningful for the study design, that the normal likelihood is a reasonable approximation at the relevant scale, and that the prior’s center and spread are defensible for the genes being modeled. If \(\tau\) is estimated from data, its estimation procedure matters. Estimating it from a set already restricted to extreme markers can distort the estimated distribution of effects; treating \(\tau\) as known can hide this uncertainty.

• Inspect the selection rule. Record whether selection used a signed contrast, absolute contrast, test statistic, adjusted p-value, or a combination. These rules imply different selection effects.
• Check standard errors and model fit. A normal-means model cannot rescue standard errors that fail to reflect the experimental units, dependence, or uncertainty in the contrast.
• Assess the prior and its fit. A zero-centered prior may be inappropriate if effects are systematically nonzero, and one common variance may be inadequate when genes have distinct levels of variability.
• Compare raw and shrunken estimates. A large reduction concentrated among imprecise extreme estimates is plausible; a result dominated by a few genes or by a small change in the assumed prior spread calls for sensitivity analysis.
• Distinguish estimation from evidence. A shrunken point estimate alone does not provide a calibrated p-value, confidence interval, or guarantee of false-discovery-rate control.

The main alternatives address different parts of the problem. Sample splitting uses one part of the data to select markers and another to estimate or test them, reducing reuse of the same noise for both tasks at a cost in precision. Independent-dataset validation checks whether effects recur in new data and is especially valuable for assessing generalizability. Selection-aware methods instead model the selection event when making inferential claims. The earlier material on selective inference and on independent-dataset validation develops those approaches; neither is interchangeable with simply shrinking a point estimate.

Shrinkage is most useful when the goal is to stabilize many related effect estimates under a credible shared model. It is limited when the selection process is complex or poorly recorded, when the prior is unreliable, or when the scientific claim requires valid uncertainty after selection. In those settings, report the selection rule and the shrinkage assumptions explicitly, and use a design or inferential method suited to the claim.

Common Mistake