Log Fold Change Bias In Selected Markers

Why Selected Log Fold Changes Look Too Large

A marker gene is often reported because its estimated expression difference between clusters is large. That selection rule creates a problem even when each gene’s estimate is unbiased before selection: estimates with unusually large errors are more likely to cross the selection threshold or rank first. Consequently, the selected log fold changes tend to exaggerate the underlying differences. This is the winner’s curse, considered here specifically for marker effect sizes.

The issue concerns the magnitude of the reported effect, not just whether a gene passes a test. A large estimated fold change can be useful for describing a contrast, but if the same noisy estimate helped select the gene, its magnitude is not an unbiased summary of the underlying effect. The distinction matters when ranking markers, planning follow-up experiments, or communicating how strongly a gene distinguishes two clusters.

In practice, expression means and their log ratios are estimated from normalized data, often using a model suited to the count structure. The argument below does not require one particular estimation procedure. It uses a local approximation in which an estimate has a roughly normal sampling distribution:

where \(\beta_g\) is the underlying log2 fold change and \(s_g\) is the standard error. This approximation is useful when the estimate is sufficiently regular and its standard error is reasonably estimated. It is not an assertion that every log ratio of sample means is exactly normal, particularly with sparse expression or very small groups. Selection bias can also occur outside the normal approximation; the model makes its size transparent.

How Threshold Selection Creates Bias

Suppose a gene is reported only if its estimate exceeds a positive cutoff \(c\), so selection occurs when \(\hat{\beta}_g>c\). Under the approximate normal model, define \(Z=(\hat{\beta}_g-\beta_g)/s_g\), which has an approximate standard normal distribution before selection. The selection condition becomes \(Z>a\), where \(a=(c-\beta_g)/s_g\). Conditional on selection, the mean estimation error is no longer zero.

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. To justify the last equality, the conditional mean is the ratio of the tail first moment to the tail probability. Since \(\phi'(z)=-z\phi(z)\),

The quantity \(\phi(a)/(1-\Phi(a))\) is positive, so selecting on a positive estimate makes the expected selected estimate exceed the true effect. When the true effect is zero and the cutoff is two standard errors, the selected estimates average more than two standard errors above zero. Selection does not make every selected estimate wrong by the same amount; it shifts the average among selected estimates upward.

A rule that selects large effects in either direction has the same issue for absolute magnitude. For a gene with true effect zero, selecting when \(|\hat{\beta}_g|>c\) makes the selected absolute estimates large on average, and the sign of the reported estimate is the sign of the noise. If true effects are nonzero, selection still tends to favor estimates whose errors point away from zero. The size of the bias depends on the true effect, the standard error, and the selection rule.

Selecting the Largest Marker

Thresholding is not the only selection mechanism. If a researcher ranks many genes and reports the largest estimated fold change, the winner is selected partly because it had a favorable estimation error. Even if every gene has the same true effect, the maximum estimate will generally exceed that common effect.

For a simple calculation, suppose \(m\) genes have the same true effect \(\beta\), independent estimates, and the same known standard error \(s\), with \(\hat{\beta}_g=\beta+sZ_g\) and independent \(Z_g\sim N(0,1)\). Then the largest estimate has expectation

The expected maximum standardized error is positive when \(m>1\). Its value grows with the number of candidates. In a real marker analysis, genes may have different standard errors, different true effects, and correlated estimates; this simple expression then does not give the exact bias. It still shows why ranking and reporting only the winner is not equivalent to reporting a gene specified before observing the data.

Effect-Size Correction Is Not Test Validation

A correction to an estimated magnitude and a valid hypothesis test answer different questions. Shrinkage can pull noisy estimates toward a reference value, often zero, reducing the expected exaggeration of selected effects under a suitable model. It does not, by itself, produce a p-value with its advertised null distribution after selection, nor does it automatically give a confidence interval with its advertised coverage for selected genes.

For example, under a normal sampling model \(\hat{\beta}_g\mid\beta_g\sim N(\beta_g,s_g^2)\) and a normal prior \(\beta_g\sim N(0,\tau^2)\), the posterior mean is a shrinkage estimate:

This formula relies on the stated prior and sampling model; in practice, \(\tau\) must be chosen or estimated. Shrinkage is a model-based effect estimate, not a universal subtraction from every selected log fold change. A valid test must separately account for how the gene was chosen, or use data and a design that make the test independent of that selection. The earlier material on selective inference and independent validation addresses those testing strategies.

Diagnostics, Comparisons, and Limitations

The selection rule should be made explicit before interpreting a reported magnitude. Ask whether genes were chosen by a fold-change cutoff, a test statistic, a ranking, or a combination of expression filtering and statistical criteria. Also establish whether the estimate and selection used the same cells and measurements. When they did, an unqualified statement that the selected estimate is the marker’s expected effect is generally too strong.

Useful checks include comparing selected estimates with estimates from an independent dataset, examining how reported magnitudes change under a prespecified split of the data, and reporting both the selection criterion and the effect-size estimator. Large changes across independent samples or splits are consistent with substantial sampling variability, though they do not alone diagnose its cause. Dependence among genes, unequal standard errors, sparse counts, and uncertain normalization can all make simple normal calculations poor quantitative approximations.

The main approaches have different targets and assumptions:

No single correction is uniformly best. Shrinkage depends on the model for effects and estimation uncertainty. Selective inference depends on describing the selection event adequately. Independent validation avoids reusing the same random fluctuation for selection and evaluation, but a validation dataset can still be small or differ from the discovery population. In all cases, the reported log fold change should be tied to its group contrast, normalization, and units.

Common Pitfall

Wrong reasoning: “The marker passed a stringent significance threshold, so its estimated fold change is an unbiased measure of how strongly it distinguishes the clusters.” A small p-value and a large estimated effect do not make the selection-dependent magnitude unbiased. A test statistic can be large because the estimate is large, because its standard error is small, or both; choosing genes based on the result affects what can be inferred from the same data.

Correction: Report the estimate as an observed, selected effect unless a justified correction or independent estimate is used. State how markers were selected, distinguish the effect-size procedure from the testing procedure, and avoid interpreting a shrinkage estimate as proof that the p-value or interval has been selection-adjusted.