Discovery–Evaluation Sample Overlap Bias

Why Shared Participants Matter

A polygenic score is often built by estimating variant effects in a discovery sample and then evaluating the resulting weighted score in a separate sample. The separation matters: if participants occur in both samples, their genotypes and phenotypes help determine the weights and also contribute to the reported evaluation. This dependence can make the score appear more strongly associated with the outcome than it would be in an independent evaluation sample.

The effect is not limited to a residual in the discovery phenotype. With marginal variant-by-variant regression, an estimated effect also contains sampling contributions from the other variants in the phenotype model. Those cross-variant contributions matter when deriving the bias. The result below assumes independent variants; linkage disequilibrium or other departures from the stated model require a different calculation.

Definition: The discovery sample is the sample used to estimate the variant weights, and the evaluation sample is the sample used to assess the score. Let $n_D$ and $n_E$ be their respective sample sizes, and let $m$ be the number of participants shared by both. Discovery–evaluation sample overlap bias is the difference between the expected evaluation statistic under this shared-sample design and its expectation under independent discovery and evaluation samples, arising from that sharing.

Model and Assumptions

Consider $M$ genetic variants and a quantitative outcome $Y$. For participant $i$, let $X_{ij}$ be the standardized value of variant $j$, with mean zero and variance one. Assume the variants are mutually independent, and use the linear model

Y_i = \sum_{j=1}^{M}\beta_j X_{ij}+\varepsilon_i, \qquad E(\varepsilon_i)=0, \qquad \operatorname{Var}(\varepsilon_i)=\sigma^2.

Assume the residuals are independent of the variants, independent across participants, and have common variance $\sigma^2$. Participants are independent of one another. Define the genetic variance in this model as $V_G=\sum_{j=1}^{M}\beta_j^2$; therefore $\operatorname{Var}(Y)=V_G+\sigma^2$. Effects and variances are treated as fixed population quantities.

For each variant, estimate its effect by a separate marginal linear regression of $Y$ on that variant, with an intercept. Under the independent-variant assumption, the population marginal regression coefficient for variant $j$ is $\beta_j$. For the leading-order calculation, approximate the centered sample denominator by $n_D$; this is appropriate as a first-order large-sample approximation when the sample variance of each standardized variant is close to one. The evaluation statistic considered first is the empirical score–outcome covariance, using the discovery estimates as weights. Centering and the distinction between $n$ and $n-1$ affect finite-sample details but not the leading overlap term derived here.

Model scope: The formula below is for marginal effect estimates under the stated independent-variant, linear-outcome model. It is not a general formula for correlated variants, non-linear outcomes, or every possible score-construction procedure.

Where the Effect-Estimation Error Comes From

Write the discovery estimate for variant $j$ as $\hat{\beta}_j=\beta_j+\delta_j$. The first-order expression for its error is

\delta_j \approx \frac{1}{n_D}\sum_{k\in D}X_{kj} \left(\varepsilon_k+\sum_{\ell\ne j}\beta_\ell X_{k\ell}\right).

The first term inside parentheses is the residual contribution. The second collects the effects of the other variants: even though variant $j$ is regressed on its own, its finite-sample estimate is affected by the other variants’ contributions to $Y$. The approximation follows by writing the marginal regression numerator as $\sum_k X_{kj}Y_k$, substituting the outcome model, and dividing by a sample sum of squares close to $n_D$.

The evaluation score for participant $i$ is $S_i=\sum_j\hat{\beta}_jX_{ij}$. Its population covariance with $Y$, if the weights were fixed and the evaluation observations independent of their estimation, would be $\sum_j\hat{\beta}_j\beta_j$. With overlapping participants, the empirical covariances between $X_j$ and $Y$ in evaluation are correlated with the discovery errors $\delta_j$. To find the leading contribution, examine the product of one participant’s discovery estimating term and evaluation contribution.

Deriving the Leading Bias

Let $C_E$ be the empirical covariance between the evaluation scores and outcomes. Ignoring centering corrections that are of smaller order in this derivation, it is $\sum_j\hat{\beta}_j\hat{c}_{jE}$, where $\hat{c}_{jE}=n_E^{-1}\sum_{i\in E}X_{ij}Y_i$. The term involving $\beta_j$ has expectation $\sum_j\beta_j^2=V_G$. The additional expectation comes from $\delta_j\hat{c}_{jE}$. For a discovery participant who is also in evaluation, its contribution involves the expectation $E[X_j(\varepsilon+\sum_{\ell\ne j}\beta_\ell X_\ell)X_jY]$. Independence, unit variances, and the model give

\begin{aligned} E\left[ X_j\left(\varepsilon+\sum_{\ell\ne j}\beta_\ell X_\ell\right)X_jY \right] &= \sigma^2+\sum_{\ell\ne j}\beta_\ell^2. \end{aligned}

In the residual part, the only nonzero contribution is $E[X_j^2\varepsilon Y]=\sigma^2$. In the other part, for each $\ell\ne j$, independence gives $E[X_j^2X_\ell Y]=\beta_\ell$, so multiplying by $\beta_\ell$ contributes $\beta_\ell^2$. A shared participant enters the discovery average with factor $1/n_D$ and the evaluation average with factor $1/n_E$. Summing over the $m$ shared participants and all $M$ variants yields the result.

Result: Under the stated model and first-order approximation, $$ E(C_E)\approx V_G+B_{\mathrm{overlap}}, \qquad B_{\mathrm{overlap}}= \frac{m\left[M\sigma^2+(M-1)V_G\right]}{n_Dn_E}. $$ The term $M\sigma^2$ is the residual contribution across $M$ marginal estimates. The term $(M-1)V_G$ is the contribution from the other variants in each marginal estimate. If the discovery and evaluation samples are independent, $m=0$ and this overlap contribution is zero.

This is a leading-order expectation, not an exact finite-sample identity. Its interpretation also depends on using marginal regression as specified. For joint regression on all variants, the leading coefficient error has a residual-only form under the corresponding model, and the cross-variant contribution above does not apply in the same way. The regression procedure must therefore be stated before using an overlap formula.

Worked Example: Calculating the Bias With Two Variants

Suppose $M=2$, $\beta_1=0.2$, $\beta_2=0.3$, and $\sigma^2=0.87$. The discovery sample has $n_D=1{,}000$ participants, the evaluation sample has $n_E=500$, and $m=100$ participants are shared. Assume the model conditions above hold.

First, $V_G=0.2^2+0.3^2=0.13$. The leading overlap contribution is

B_{\mathrm{overlap}} = \frac{100\left[2(0.87)+(2-1)(0.13)\right]}{1{,}000(500)} = \frac{187}{500{,}000} = 0.000374.

Thus the expected evaluation score–outcome covariance is approximately $0.13+0.000374=0.130374$, compared with $0.13$ under independent samples, at this order. Since the variants are standardized, the score is on the outcome scale and this covariance has outcome-variance units. The added amount is small relative to $V_G$ in this example, but its direction is upward.

Worked Example: Why the Cross-Variant Term Cannot Be Dropped

Let $M=2$, $\beta_1=0$, and $\beta_2=0.4$, with $\sigma^2=0.84$. Let $n_D=100$, $n_E=50$, and $m=1$. Then $V_G=0.16$. The full calculation gives

B_{\mathrm{overlap}} = \frac{1\left[2(0.84)+(2-1)(0.16)\right]}{100(50)} = \frac{1.84}{5{,}000} = 0.000368.

For the shared participant, the first-order error in the marginal estimate of $\beta_1$ includes $0.4X_1X_2/100$. Its product with that participant’s evaluation contribution $X_1Y/50$ has expectation $0.4^2/(100\cdot50)=0.000032$, because $E[X_1^2X_2Y]=0.4$. This is exactly the cross-variant part of the total bias in this example. A residual-only calculation would give $2(0.84)/5{,}000=0.000336$, omitting $0.000032$ and understating the leading covariance contribution.

From Covariance to Prediction Metrics

Covariance inflation can affect correlation-based performance measures, but it is important not to equate the bias in one numerator with an exact bias formula for every metric. A score–outcome correlation divides covariance by the product of the score and outcome standard deviations. Those denominators can also be affected by estimated weights and finite evaluation samples. For example, under the first-order model, the sum of marginal-estimation error variances is approximately $[M\sigma^2+(M-1)V_G]/n_D$. Thus using the covariance shift alone as an exact formula for expected $R^2$, correlation, or prediction error would be unjustified.

A useful approximation, when the relevant variances are treated as fixed, is that the correlation changes in proportion to the covariance. If $V_S$ and $V_Y$ denote the score and outcome variances used in this approximation, the squared-correlation proxy is $C_E^2/(V_SV_Y)$. In an actual evaluation, estimate or account for both variances and use a design that supports the intended interpretation. Prediction metrics can still look better with overlap because the score weights have partly adapted to outcomes of people whose outcomes are being used for evaluation; the covariance result identifies a direct mechanism, not a universal correction for every reported metric.

Worked Example: Interpreting a Covariance Shift Alongside a Metric

Suppose $M=100$, $V_G=0.20$, $\sigma^2=0.80$, $n_D=1{,}000$, $n_E=500$, and $m=250$. The outcome variance is one. The leading covariance contribution is

B_{\mathrm{overlap}} = \frac{250\left[100(0.80)+99(0.20)\right]}{1{,}000(500)} = \frac{24{,}950}{500{,}000} = 0.0499.

The expected covariance is therefore approximately $0.20+0.0499=0.2499$, about 25.0% above the non-overlap covariance $0.20$. For scale only, the first-order sum of marginal-estimation error variances is $[80+99(0.20)]/1{,}000=0.0998$; adding $V_G$ gives an approximate score variance of $0.2998$. Holding that variance and the outcome variance at one, the ratio-of-moments squared-correlation proxy is $0.2499^2/0.2998\approx0.2083$, versus $0.20^2/0.2998\approx0.1334$ using the non-overlap covariance. These are illustrative proxies, not exact expected sample $R^2$ values: an empirical performance analysis must address the random denominators and sampling variation as well as the numerator.

Design Remedies and a Common Pitfall

The most direct remedy is design-based: keep the evaluation participants out of discovery, and keep close relatives out as well when their shared genetic and outcome information violates the intended independence. A genuinely independent evaluation cohort makes $m=0$ and removes this particular overlap contribution under the model. If a completely independent cohort is unavailable, possible design choices include excluding shared participants from one stage, using non-overlapping discovery and evaluation splits, or using cross-fitting so that each participant’s score is estimated without that participant’s data. Report the sample sizes and the number of participants shared; the overlap fraction alone is not enough to determine the formula because both $n_D$ and $n_E$ enter.

Common mistake: “Only the residual appears in an effect-estimation error, so the overlap bias is $Mm\sigma^2/(n_Dn_E)$ for marginal estimates.” That reasoning drops the sampling terms from the other variants. In a marginal regression, the error for variant $j$ includes $X_j\sum_{\ell\ne j}\beta_\ell X_\ell$ as well as $X_j\varepsilon$. Their correlation with evaluation genotype–outcome products adds $m(M-1)V_G/(n_Dn_E)$. The residual-only expression is not the stated marginal-regression result.

Specify the estimation procedure.
State whether weights come from marginal regressions or a joint model; the error terms differ.

Count the overlap.
Record $n_D$, $n_E$, and $m$, using the same participant definition in both samples.

Interpret the evaluation metric cautiously.
The derived bias applies to the leading score–outcome covariance under the model. Other performance measures also depend on their denominators and sampling design.

Key takeaway: For marginal estimates under independent standardized variants, shared participants inflate the leading expected evaluation score–outcome covariance by $m[M\sigma^2+(M-1)V_G]/(n_Dn_E)$. Independent evaluation removes this overlap term; a residual-only calculation does not describe marginal regression when genetic effects are present.

Check Your Understanding

Answer each question using the model and assumptions stated above.

Which two components of marginal effect-estimation error contribute to the overlap formula?
If $m=0$, what is the leading overlap contribution to the expected evaluation covariance?
For $M=3$, $V_G=0.12$, and $\sigma^2=0.88$, identify the residual and cross-variant parts of the numerator $M\sigma^2+(M-1)V_G$.
Why does the residual-only expression fail for marginal regression when $V_G>0$?
Why is the covariance result not, by itself, an exact correction formula for expected evaluation $R^2$?