Why Threshold Selection Changes a Test
Suppose a gene is reported as a marker only if its test statistic exceeds a prespecified cutoff. Testing that gene as though it had been chosen independently ignores how it entered the reported set: even under a null effect, statistics that pass the cutoff are unusually large. The relevant null distribution for a reported statistic is therefore not the original Gaussian distribution, but that distribution conditional on exceeding the cutoff.
The earlier material on selective inference conditional on cluster selection motivates conditioning on the event that led to a result being examined. Here the selection event is narrower and explicit: one Gaussian statistic is retained only when it exceeds a fixed threshold. The calculation below adjusts for that event; it does not, by itself, adjust for ranking many genes or for other data-dependent selection steps.
Write \(Q(x)=P(N(0,1)\ge x)=1-\Phi(x)\), where \(\Phi\) is the standard normal cumulative distribution function. Because \(z_{\mathrm{obs}}>c\), the event \(Z\ge z_{\mathrm{obs}}\) is contained in \(Z>c\). The conditional probability is consequently the probability of reaching the observed value divided by the probability of passing the selection threshold:
This is an upper-tail test: large values count against \(H_0\). The selective p-value measures how unusual the observed statistic is among statistics that would have been selected by this threshold, assuming the null model. It is not the probability that the null is true, nor the probability that the gene is a true marker.
Why the Tail-Ratio Formula Works
Under \(H_0\), the density of \(Z\) before selection is the standard normal density \(\phi(z)\). Conditioning on \(Z>c\) restricts its support to \(z>c\) and divides the density by the probability of that event, \(Q(c)\). Thus the conditional density is:
To obtain the conditional upper-tail probability at \(z_{\mathrm{obs}}\), integrate this density from the observed value to infinity. Since \(z_{\mathrm{obs}}>c\), the integration range is entirely within the selected region:
This derivation also explains the calibration: if the null model and selection event are correct, the conditional p-value is uniform on \([0,1]\) among selected statistics under \(H_0\). To see this, let \(U=Q(Z)/Q(c)\) for \(Z>c\). For any \(0\le u\le1\), the threshold \(x\) satisfying \(Q(x)=uQ(c)\) is at least \(c\), and \(P_0(U\le u\mid Z>c)=P_0(Z\ge x\mid Z>c)=Q(x)/Q(c)=u\). Thus small selective p-values occur at their nominal rates within the selected null statistics. This justification depends on conditioning on the event actually used for selection, not merely on an event that resembles it.
Worked Calculations
Worked Example: A Statistic Just Beyond a Conventional Cutoff
Hypothetical data. A one-sided gene statistic is \(z_{\mathrm{obs}}=2.40\). The analysis reports the gene only if \(Z>1.645\), with that threshold set in advance. Assume that under the point null, \(Z\sim N(0,1)\), and that crossing this threshold is the only selection event relevant to this test.
The selective p-value is the null probability of reaching 2.40 divided by the null probability of passing 1.645:
The ordinary, unconditioned one-sided p-value is \(Q(2.40)\approx0.008198\). That value answers a different question: it does not account for the rule that the statistic had to exceed 1.645 to be reported. Conditional on passing that rule, a value at least as large as 2.40 has probability about 0.1640 under the null. This is not small evidence against \(H_0\) at \(\alpha=0.05\).
Worked Example: A More Stringent Selection Threshold
Hypothetical data. A gene statistic is retained only if it exceeds \(c=2.00\). For one retained gene, \(z_{\mathrm{obs}}=2.88\). Assume a standard normal null statistic and no additional selection beyond this fixed cutoff.
Substitute both the observed value and the selection threshold into the tail ratio. Evaluating the ratio using unrounded standard normal tail probabilities gives:
The conditional p-value is greater than 0.05, so this single threshold-conditional test does not reject \(H_0\) at the 5% level. The observed statistic is still relatively high compared with the original null distribution, but it is less exceptional when compared with the distribution of statistics already known to exceed 2.00.
Worked Example: A Selected Statistic Near the Cutoff
Hypothetical data. A prespecified reporting rule retains statistics above \(c=2.326\), approximately the 99th percentile of a standard normal distribution. A retained gene has \(z_{\mathrm{obs}}=2.50\). Assume the same standard normal null model and that this cutoff fully describes the selection.
The calculation is:
Although the unconditioned one-sided p-value is \(Q(2.50)\approx0.006210\), the selective p-value is about 0.6204. This large difference arises because the threshold itself is already very stringent: among null statistics that pass it, a value of 2.50 is not unusually high. The conditional result does not say the gene has no biological relevance; it quantifies evidence under the stated null model and selection rule.
Assumptions and Scope
The ratio is exact under a specific set of conditions. Before using it, check each of the following:
- Gaussian null distribution. The statistic must have the stated \(N(0,1)\) distribution under \(H_0\), or a justified Gaussian model that can be standardized to it. A statistic with a different null distribution needs its own conditional tail calculation.
- Known and appropriate calibration. Any standardization used to obtain \(Z\) must be valid under the null. If variance estimates, asymptotic approximations, or nuisance parameters materially affect the null distribution, the standard normal tail ratio may only be approximate or may be invalid.
- Fixed cutoff. The threshold \(c\) must be set independently of the observed statistic being tested. Choosing or tuning it after inspecting the same values changes the selection event.
- Selection is exactly represented by \(Z>c\). If the gene was also chosen because it ranked first, passed another filter, or had a favorable result in another analysis, conditioning only on this threshold does not account for those steps.
- One-sided upper-tail selection and testing. The selection rule and alternative here concern large positive values. Selection by \(|Z|>c\), selection on a lower tail, or a two-sided claim requires a different conditional event and tail calculation.
For a statistic \(T\) with null distribution \(N(\mu_0,\sigma^2)\), the same reasoning applies after standardization. If selection is \(T>t_0\) and \(t_{\mathrm{obs}}>t_0\), then:
Here \(\mu_0\) and \(\sigma\) are the null mean and standard deviation, not quantities chosen to make the observed result look more or less significant. In the usual standardized one-sided test, \(\mu_0=0\) and \(\sigma=1\), yielding the formula used in the examples.
Diagnostics, Alternatives, and Common Mistakes
A useful first diagnostic is procedural: write down the selection rule in words and symbols before calculating. Confirm that the threshold was fixed, that the reported statistic really passed it, and that no other step determined which result was presented. Next, check the null calibration and scale of the statistic. A ratio of Gaussian tails cannot repair a misspecified null distribution, dependence-induced miscalibration, or a cutoff that does not describe the actual selection.
The main alternative is to obtain the test statistic from data that were not used to select the gene or define the tested groups. Sample splitting and independent validation can avoid conditioning on the same selection event for the confirmatory test, though they may have less precision or require additional data. A naive test using \(Q(z_{\mathrm{obs}})\) is simpler, but it is not a valid replacement when the same statistic was selected by the stated threshold and the goal is inference conditional on selection. Conversely, this single-gene tail ratio does not account for searching across many genes or selecting the largest statistic; those procedures have more complex selection events.
A common mistake is to argue, “The raw p-value is below 0.05, so the selected gene is significant,” without considering how it entered the reported set. In the first worked example, \(Q(2.40)\approx0.008198\), but the threshold-conditional value is approximately 0.1640. The correction is not to multiply by an arbitrary factor; it is to calculate the null tail probability conditional on the actual selection event.
Check Your Understanding
Answer each question using the upper-threshold model above. Full solutions follow the questions.
- A standard normal statistic is selected when \(Z>1.96\) and the observed value is 2.326. Write the selective p-value and give its approximate value.
- A statistic is selected when \(Z>1.5\), and the observed value is exactly 1.5. What is the limiting threshold-conditional p-value?
- A reporting rule requires \(Z>2\), but the observed statistic is 1.8. Can the selected-statistic formula be applied to this observation? Explain.
- For selection at \(Z>1\) and an observed value of 2, calculate the selective p-value and distinguish its interpretation from the ordinary p-value.
- Why does the threshold correction fail to account for a gene that was reported because it had the largest statistic among hundreds of genes?
Solutions. (1) The expression is \(Q(2.326)/Q(1.96)\). Since these tails are approximately 0.0100 and 0.0250, respectively, the ratio is approximately \(0.400\). This is the upper-tail probability conditional on passing 1.96. (2) At the boundary, the conditional tail probability is \(Q(1.5)/Q(1.5)=1\); if selection is defined strictly as \(Z>1.5\), the same value is approached as the observed statistic decreases to the threshold from above. (3) No. The observation did not pass the stated selection rule, so it is not a selected statistic under this model. One should check whether the rule or the reported value has been misstated. (4) The calculation is \(Q(2)/Q(1)\approx0.0227501/0.1586553\approx0.1434\). The ordinary p-value, about 0.02275, is the null upper-tail probability without conditioning; 0.1434 is the probability of reaching 2 among null statistics that already exceed 1. (5) Ranking among many genes is an additional selection event. Conditioning only on \(Z>c\) ignores the fact that other statistics were examined and that the reported one won the ranking; the tail ratio therefore does not represent the full conditional null distribution for that selection process.