Deriving Selection Bias After Significance Thresholding

Why a Significance Threshold Changes the Estimate

A genetic association estimate varies from sample to sample, even when the underlying effect is fixed. If researchers report an estimate only when its test statistic crosses a significance threshold, the reported estimates are drawn from a restricted part of their sampling distribution. Estimates near the true effect that happen to be unusually large are more likely to pass the filter than estimates that happen to be small. The conditional mean among selected results can therefore differ from the underlying effect.

This tutorial derives that difference under a normal sampling model. The key distinction is what was selected: a result exceeding a threshold in a prespecified positive direction, or a result exceeding a threshold in either direction. The latter selects on the absolute test statistic. These events are not interchangeable: under a zero true effect, two-sided selection leaves the mean signed estimate at zero by symmetry, even though the selected absolute estimates are large on average.

Model, Assumptions, and Definitions

Let \(\hat{\beta}\) estimate a genetic association effect \(\beta\), measured in the study's effect units. Assume its standard error \(SE\) is fixed and known for this calculation and that \(\hat{\beta}\sim N(\beta,SE^2)\). Define the standardized estimate \(Z=\hat{\beta}/SE\) and the standardized true effect \(\delta=\beta/SE\). Then \(Z\sim N(\delta,1)\). Let \(\phi\) and \(\Phi\) denote the standard normal density and cumulative distribution functions, and let \(Q(x)=1-\Phi(x)\).

The formulas below require the normal model, a fixed \(SE\), and a fixed threshold \(c\). They describe selection for one estimate under that threshold. If the standard error is estimated with substantial uncertainty, the test statistic is not well approximated by this fixed-variance normal model. Likewise, selecting the largest result among many correlated tests involves an additional selection rule not captured solely by conditioning one statistic on \(|Z|>c\).

Positive-Direction Selection

For a prespecified positive-direction test, the probability of selection is \(P(Z>c)=Q(c-\delta)\). To obtain the conditional mean, write \(Z=\delta+\varepsilon\), where \(\varepsilon\sim N(0,1)\). The normal tail identity \(E[\varepsilon\,1(\varepsilon>a)]=\phi(a)\) follows by integrating \(u\phi(u)\) from \(a\) to infinity, since \(\phi'(u)=-u\phi(u)\). With \(a=c-\delta\), this gives

Thus the conditional mean exceeds \(\delta\): the added term is positive. Returning to the effect scale, \(\hat{\beta}=SE\cdot Z\) and \(\beta=SE\cdot\delta\), so

The amount of conditional inflation depends on both the threshold and the true standardized effect. It is not a fixed correction that can be applied to every selected estimate.

Two-Sided Selection: Signed Estimates and Absolute Magnitudes

For selection by \(|Z|>c\), both tails contribute. The selection probability is \(Q(c-\delta)+\Phi(-c-\delta)\). The positive-tail contribution to the unnormalized signed expectation is \(E[Z\,1(Z>c)]=\delta Q(c-\delta)+\phi(c-\delta)\). The negative-tail contribution is \(E[Z\,1(Z<-c)]=\delta\Phi(-c-\delta)-\phi(c+\delta)\). Adding these contributions and dividing by the selection probability yields the signed conditional mean:

For the absolute magnitude, the negative-tail contribution changes sign because \(|Z|=-Z\) there. Therefore,

Multiplying either conditional expectation by \(SE\) gives the corresponding estimate in effect units. Under a zero true effect, \(\delta=0\), the signed formula is zero: the selected positive and negative estimates cancel. The absolute formula instead simplifies to

Why the Distinction Matters

A two-sided \(p\)-value threshold selects on \(|Z|\), not on \(Z\) alone. If the effect direction was specified before looking at the data, positive-direction selection can be represented by \(Z>c\) (or by the corresponding negative tail for a prespecified negative direction). If the direction is chosen after examining the estimate, the selected magnitude is relevant, and treating the event as a prespecified positive-direction test gives the wrong conditional expectation.

The difference is especially clear under the null. Two-sided significance filtering does not make the mean signed estimate positive when both signs are retained; it makes the mean absolute estimate positive and large relative to the threshold. If a report flips signs so every selected estimate points in the observed direction, it has changed the quantity being summarized from a signed estimate to a magnitude.

In a genomic analysis, the threshold may be very stringent, and the number of tested variants may be large. These formulas explain the conditional effect of crossing a specified threshold for a given statistic. They do not by themselves account for choosing a lead variant among many correlated candidates, changing the analysis after seeing results, or uncertainty in the estimated standard error. Those features require a model of the actual selection process.

Diagnostics, Alternatives, and Limitations

Before using a conditional expectation, write down the selection event in the same terms as the statistic. Check whether the direction was prespecified, whether the cutoff is one- or two-sided, and whether the reported estimate retains its original sign. Confirm that \(\hat{\beta}\) and \(SE\) are on the same scale and that \(Z=\hat{\beta}/SE\) is the statistic being thresholded. A mismatch between the reported selection rule and the event in the calculation invalidates the result.

Useful warning signs include interpreting \(E[|Z|\mid |Z|>c]\) as a signed mean, treating a two-sided \(p\)-value filter as positive-tail selection, or applying the one-test formula to a lead estimate selected as the largest among many loci. The normal model is also a poor approximation if the estimator distribution is strongly skewed or if \(SE\) is highly uncertain. In those cases, the selection mechanism and sampling distribution need to be represented more fully.

The formulas here derive and quantify selection bias; they are not correction procedures. In particular, a conditional mean is not enough to recover an individual locus's true effect. A correction must also specify how the selection event and the information in the observed estimate are used; that is a separate estimation problem.

Common Mistake