Two-Tail Extreme Sampling and Effect Interpretation

Why the Direction of Phenotype Selection Matters

Sequencing people at the extremes of a quantitative phenotype can be an efficient way to find rare-variant associations: if a variant shifts the phenotype, carriers may be more common among people with unusually high or low values than among people near the center. But “extreme” can describe different designs. Investigators might select only the upper tail, only the lower tail, both tails and compare them, or pool both tails into one extreme group. These choices can produce different carrier contrasts and support different interpretations.

The key distinction is between enrichment and direction. Pooling both tails can enrich carriers when a variant moves the phenotype away from the center, whether that movement is upward or downward. But that pooled enrichment alone does not reveal which direction the variant shifts the phenotype. Comparing the upper and lower tails retains directional information, although the resulting contrasts are shaped by selection and are not automatically estimates of the source-population effect.

This is a deliberately simple model for deriving the consequences of selection. In an analysis, ancestry, covariates, relatedness, phenotype transformations, and other design features may also matter. The formulas below describe the model as stated; they do not remove the need to address those features in a real study.

Selection Probabilities and Carrier Enrichment

Let \(\Phi(x)\) denote the standard normal cumulative distribution function, let \(\phi(x)\) denote its density, and write \(Q(x)=1-\Phi(x)\). Choose a positive threshold \(a\), measured in residual standard deviations. The upper-tail group has \(Y>\mu+a\sigma\); the lower-tail group has \(Y<\mu-a\sigma\); the symmetric two-tail group has either condition.

Conditional on genotype \(G=g\), the standardized phenotype \((Y-\mu)/\sigma\) has a normal distribution with mean \(\delta g\) and variance 1. Therefore, the probabilities of entering each selected group are

and for selection into either tail,

These are probabilities of selection within each genotype group, not carrier proportions within the selected samples. To obtain the latter, apply Bayes’ rule. For example, the carrier proportion among upper-tail participants is

The corresponding lower-tail or pooled two-tail proportion is found by replacing \(q_g^+\) with \(q_g^-\) or \(q_g^{\pm}\). A small source carrier proportion can still produce substantial enrichment if carriers are much more likely to enter a selected group. Conversely, a strong relative enrichment does not imply that most selected people are carriers.

Why Either Tail Can Enrich Carriers

A symmetric two-tail design selects people whose phenotypes are far from the center in either direction. Under the model, the probability that a carrier enters that group is

The function is symmetric in the effect direction: \(T(-\delta)=T(\delta)\). One way to see this is to substitute \(Q(x)=\Phi(-x)\) and use the symmetry of the standard normal distribution. Its derivative is

For \(\delta>0\), \(|a-\delta|<a+\delta\), so the standard normal density at \(a-\delta\) is greater than the density at \(a+\delta\). Consequently, \(T'(\delta)>0\). By symmetry, \(T(\delta)\) also increases as a negative effect moves farther below zero. Thus, for any nonzero shift, carriers are more likely than noncarriers to be in the pooled extreme group under this model. This gives the mathematical reason that a two-tail pooled design can detect departure from the center without identifying the sign of the effect.

If the upper and lower tails are retained as separate groups, their contrast does carry directional information. For a positive \(\beta\), carriers are relatively enriched in the upper tail and depleted in the lower tail; for a negative \(\beta\), the pattern reverses. The expected carrier-proportion difference between the two selected groups is a design-specific contrast, not a direct reading of \(\beta\) in phenotype units.

Justifying the Truncated-Mean Formula

The conditional means above follow directly from the normal density. For \(Z\sim N(0,1)\), the identity \(\phi'(z)=-z\phi(z)\) gives

The probability of the same upper-tail event is \(Q(c)\), so dividing the integral by that probability gives \(E(Z\mid Z>c)=\phi(c)/Q(c)\). By symmetry, \(\int_{-\infty}^{-c}z\phi(z)\,dz=-\phi(c)\), and the lower-tail probability is \(Q(c)\). This yields \(E(Z\mid Z<-c)=-\phi(c)/Q(c)\). Applying these results to the carrier-specific residual cutoffs gives the conditional-mean formulas used above.

The derivation also explains why the selected mean contrast should not be interpreted as the source effect. Selection changes the distribution of the residuals among included people, and it changes that distribution differently for carriers and noncarriers.

Comparing the Main Design Choices

These distinctions matter when planning sequencing and interpreting gene-level results. A burden comparison between separately sampled high and low groups targets a directional contrast in that selected design. A pooled extreme-versus-center comparison targets a different question: whether carriers are overrepresented among extreme phenotypes. A test designed for one contrast should not be described as though it estimated the other.

Design and Interpretation Pitfalls

The most common error is to treat a carrier proportion in a selected sample as if it were the source-population carrier prevalence, or to treat a selected-sample phenotype contrast as if it were the unselected effect. The correct reasoning is to model how likely each genotype is to enter each selected group, then interpret the observed contrast for that design. Under the model, the carrier proportion is a function of both the source prevalence \(p\) and the genotype-specific selection probabilities.

Another important issue arises when investigators sample fixed numbers from each tail. The within-tail carrier proportions still describe their respective selected groups, but the proportion after pooling those groups depends on how many people were recruited from each tail. If recruitment quotas do not reflect the natural numbers of people in the two tails, the pooled sample proportion is a quota-weighted quantity, not the source probability of being extreme. Any comparison that uses the pooled group must account for this sampling design.

• Wrong reasoning: “The pooled extreme group has more carriers than the source population, so the variant must increase the phenotype.” Correction: Either a sufficiently positive or sufficiently negative shift can enrich carriers in a symmetric pooled two-tail group. Separate the tails to assess direction.
• Wrong reasoning: “The carrier mean difference among selected people equals \(\beta\).” Correction: Tail selection truncates the phenotype distributions. The conditional means involve truncated-normal terms and generally differ from the source means.
• Wrong reasoning: “The carrier proportion in a deliberately assembled sequencing sample estimates \(p\).” Correction: It estimates a proportion in the sampled design. Recovering or interpreting source-population quantities requires accounting for ascertainment and recruitment weights.

These conclusions rely on the stated equal-variance normal model and on selection based only on the phenotype thresholds. If carrier and noncarrier variances differ, if selection also depends on genotype-related factors, or if population structure and covariates affect both genotype and phenotype, the displayed probabilities need modification. The general lesson remains: state the selection event explicitly before interpreting a genotype contrast.