Extreme-Phenotype Sampling for Rare-Variant Discovery

Why Select Participants From a Phenotype Tail?

Sequencing every member of a large population can be costly, while many rare-variant effects are difficult to detect in an unselected sample. If carriers tend to have more extreme values of a relevant phenotype, deliberately recruiting people from one tail can increase the proportion of carriers among those sequenced. That can make a discovery study more efficient, but it changes the sample: the people recruited are not a random cross-section of the population.

A useful design therefore has two linked questions. First, what phenotype threshold will define eligibility, and how many people must be screened to recruit the desired number? Second, under a stated model, how much does that threshold enrich carriers, and what limits does the selection impose on interpreting results for the wider population?

Set the Threshold in the Target Population

The target population must be specified before choosing a threshold. It is the population to which the study aims to speak, such as adults from a defined region and ancestry group who meet stated eligibility criteria. A threshold is meaningful only relative to a phenotype measured comparably in that population. Age, sex, medication, environment, and measurement protocol can all affect the phenotype distribution, so a cutoff derived in one setting may not select the same fraction in another.

For a planning model, suppose the phenotype in the target population follows \(Y\sim N(\mu,\sigma^2)\). A threshold \(t=\mu+c\sigma\) then has screening yield \(q=P(Y>t)=1-\Phi(c)\), where \(\Phi\) is the standard normal cumulative distribution function. In practice, the threshold may instead be set from an empirical reference distribution or a clinically meaningful criterion. The key design quantity is the resulting proportion eligible, not the particular method used to choose the cutoff.

The threshold should be prespecified and assessed with a measurement process that is consistent across screening and recruitment. If the threshold is estimated from a small or unrepresentative reference sample, its actual yield may differ from the planned yield. Repeated measurements, eligibility confirmation, nonresponse, and sequencing failures can further reduce the number of usable samples.

How a Threshold Enriches Carriers

A simple model makes the mechanism explicit. Let \(G=1\) indicate that a participant carries a variant or meets a prespecified gene-level carrier definition, and let \(G=0\) indicate noncarrier status. Suppose the carrier prevalence in the target population is \(K=P(G=1)\), and, conditional on carrier status, the phenotype follows \(Y\mid G=0\sim N(\mu_0,\sigma^2)\) and \(Y\mid G=1\sim N(\mu_0+\delta,\sigma^2)\). Here \(\delta>0\) represents a higher mean phenotype among carriers, and this model assumes equal within-group variances and a common measurement scale.

For threshold \(t\), define \(q_0=P(Y>t\mid G=0)\) and \(q_1=P(Y>t\mid G=1)\). The overall screening yield is the sum of the carrier and noncarrier contributions. Among those selected, the carrier fraction follows from the conditional probability formula.

When \(q_1>q_0\), the selected carrier fraction exceeds the population prevalence \(K\). The enrichment ratio is \(P(G=1\mid Y>t)/K\). It is not determined by the threshold alone: it also depends on the carrier prevalence and on how strongly phenotype distributions differ. For a gene with many possible rare variants, a single carrier indicator and a single \(\delta\) may be a rough planning approximation; variant effects can differ in direction and magnitude, and some carriers may have no relevant effect.

Translate Recruitment Goals Into Screening Requirements

If the probability of meeting the threshold is \(q\), then screening \(N\) eligible target-population candidates gives an expected selected count \(Nq\), provided each person has the same selection probability and screening outcomes are independent for this planning calculation. To obtain an expected count of at least \(m\), choose the smallest whole number \(N\) satisfying \(Nq\ge m\), or \(N=\lceil m/q\rceil\). This is a planning expectation, not a promise that the realized count will reach \(m\).

When carrier status is rare, the expected number of selected carriers is especially useful for study planning. If \(N_s\) participants are recruited and the model predicts selected carrier fraction \(r=P(G=1\mid Y>t)\), then the expected carrier count is \(N_s r\). That count can inform feasibility, but it does not by itself establish that a gene-level association test will have adequate power: effect heterogeneity, sequencing quality, variant annotation, population structure, and the chosen analysis also matter.

Why Selected-Sample Results Do Not Automatically Generalize

Selection on phenotype changes the distribution of both phenotype and, when carriers affect the phenotype, genotype. Consequently, the carrier fraction observed among sequenced participants estimates \(P(G=1\mid Y>t)\), not the target-population prevalence \(P(G=1)\). Likewise, a relationship estimated only within the selected tail describes that restricted range and sampling scheme; it need not equal the population-wide relationship.

The distinction is important even when the study's scientific aim is discovery rather than estimating prevalence. A high carrier yield can help identify candidate associations, but selected participants cannot be treated as a random population sample for descriptive frequency estimates. Inferences about effects across the full phenotype range require an analysis that accounts for the sampling design and an appropriate target-population model or reference data. The needed correction depends on what was sampled and what parameter is being estimated.

For example, if the inclusion probabilities \(q_1\) and \(q_0\) in the hypothetical model were known and the model were correct, the selected carrier fraction \(r\) could be related back to the population prevalence by solving \(r=Kq_1/[Kq_1+(1-K)q_0]\) for \(K\). Rearranging gives \(K=rq_0/[q_1(1-r)+rq_0]\). In real studies these probabilities are rarely known exactly, and can depend on covariates, phenotype measurement, and recruitment processes. A selected-sample fraction alone is therefore insufficient to recover population prevalence without additional design information or assumptions.

Common Mistakes and Practical Checks

• Calling the selected carrier fraction the population prevalence. A tail sample is intentionally enriched, so its observed carrier fraction estimates the fraction among selected participants, not among all members of the target population.
• Treating expected yield as a guaranteed count. \(Nq\) is an expectation. Random variation and recruitment losses can leave the realized number below the target.
• Assuming a more extreme threshold always improves the study. A stricter cutoff can increase carrier enrichment under some effect models, but it also lowers screening yield and may produce a smaller, less representative sample. Enrichment alone is not a complete design criterion.
• Ignoring phenotype measurement and population differences. A threshold defined in one population or measurement setting may have a different screening yield elsewhere. Validate the threshold and selection rate in the intended setting.
• Assuming every carrier shifts the phenotype in the same way. Rare variants in a gene can have differing effects. A single-shift model is a transparent planning approximation, not a universal description of gene-level biology.

A robust design records the target population, phenotype definition, threshold, estimated screening fraction, and recruitment losses. It also distinguishes the goal of enriching carriers for discovery from the separate goal of estimating population frequencies or effects. Those goals can require different sampling and analysis strategies.