Combining Independent Replication Estimates With Selected Discovery

Why Selection Changes the Combination

A discovery estimate is often reported because it crossed a significance threshold. Among estimates selected this way, large values are overrepresented: even when the underlying effect is modest, sampling error can help an estimate pass the threshold. An independent replication estimate has a different role. If the replication data are independent of discovery and are not themselves selected for a favorable result, selection of the discovery estimate does not bias the replication estimate.

The practical question is not simply whether to average two estimates. It is whether the usual pooling calculation describes the sampling process that produced them. Ordinary inverse-variance pooling assumes that both estimates have their stated sampling distributions without conditioning on a selection event. That assumption fails for a discovery estimate reported only after passing a threshold. Replication-only inference can therefore provide a defensible anchor, while a combination that uses discovery information should represent the selection rule explicitly.

Definition: Let $\hat{\theta}_D$ be a discovery estimate and $\hat{\theta}_R$ an independent replication estimate of the same effect $\theta$, with standard errors $s_D$ and $s_R$. A discovery estimate is selected here if it is observed or reported conditional on a specified event $S$, such as $\hat{\theta}_D/s_D>c$. The replication estimate can anchor inference when it is independent of discovery, unbiased for the same target effect, and analyzed without selection based on its observed value.

For the calculations below, assume a common fixed effect, harmonized effect scales and alleles, independent nonoverlapping cohorts, and normally distributed estimates with known or sufficiently reliable standard errors:

\hat{\theta}_D \sim N(\theta,s_D^2), \qquad \hat{\theta}_R \sim N(\theta,s_R^2), \qquad \hat{\theta}_D \perp \hat{\theta}_R.

Before selection, both estimates have expectation $\theta$. After conditioning on discovery selection, the distribution of $\hat{\theta}_D$ changes. Independence means that conditioning on a discovery event does not change the replication distribution: $E(\hat{\theta}_R\mid S)=\theta$. This is the basic reason replication-only inference remains valid under the assumptions above. It does not make the replication estimate precise, nor does it establish that the effect is causal.

What a Threshold Does to a Discovery Estimate

Consider a one-sided selection rule $S=\{\hat{\theta}_D/s_D>c\}$, with the direction and threshold $c$ specified in advance. Define $Z=(\hat{\theta}_D-\theta)/s_D$ and $a=c-\theta/s_D$. Under the normal model, selection is equivalent to $Z>a$. The standard normal density and distribution function are denoted by $\phi$ and $\Phi$, and define the inverse Mills ratio $\lambda(a)=\phi(a)/[1-\Phi(a)]$.

Result: Under this one-sided normal-threshold model, $$ E(\hat{\theta}_D\mid S)=\theta+s_D\lambda(a), \qquad \operatorname{Var}(\hat{\theta}_D\mid S)=s_D^2\left[1+a\lambda(a)-\lambda(a)^2\right]. $$ The conditional mean exceeds $\theta$, while the conditional variance is not, by itself, a standard error for an estimate of $\theta$.

A short derivation shows where these expressions come from. Since $Z$ has a standard normal distribution, integration using $\phi'(z)=-z\phi(z)$ gives $\int_a^\infty z\phi(z)\,dz=\phi(a)$, so $E(Z\mid Z>a)=\phi(a)/[1-\Phi(a)]=\lambda(a)$. A second integration by parts gives $\int_a^\infty z^2\phi(z)\,dz=a\phi(a)+1-\Phi(a)$, hence $E(Z^2\mid Z>a)=1+a\lambda(a)$. Subtracting the squared conditional mean yields the stated variance; transforming back from $Z$ to $\hat{\theta}_D=\theta+s_DZ$ gives the result for the estimate.

Worked Example: Expected Inflation Under a One-Sided Threshold

Suppose, hypothetically, the true effect is $\theta=0.040$ standardized trait units per allele and the discovery standard error is $s_D=0.040$ units. A discovery estimate is selected when its $Z$-statistic exceeds $c=1.80$. Calculate the conditional mean and standard deviation of selected estimates under this model.

Here $a=c-\theta/s_D=1.80-0.040/0.040=0.80$. To illustrate the calculation at the $1.80$ threshold itself, consider instead selection on the uncentered statistic $\hat{\theta}_D/s_D>1.80$, as is commonly specified in a test. Then the standard-normal truncation point is $a=1.80-\theta/s_D=0.80$ only if centering at the true effect; equivalently the selected-noise threshold is $0.80$. Thus the relevant values are $\phi(0.80)\approx0.2897$, $1-\Phi(0.80)\approx0.2119$, and $\lambda(0.80)\approx1.367$.

Using the more precise $\lambda(0.80)\approx1.367402$, the conditional mean is $0.040+0.040(1.367402)\approx0.09470$ units. The variance is $0.040^2[1+0.80(1.367402)-(1.367402)^2]\approx0.0003586$ units squared, giving a conditional standard deviation of about $0.01894$ units. These are moments of the selected estimate under the assumed true effect, not a bias-corrected estimate or an uncertainty interval for an observed locus.

A different convention is to select on the centered error $Z=(\hat{\theta}_D-\theta)/s_D>1.80$. Under that rule, $a=1.80$: $\phi(1.80)\approx0.07895$, $1-\Phi(1.80)\approx0.03593$, and $\lambda(1.80)\approx2.1973$. The conditional mean is $0.040+0.040(2.1973)\approx0.1279$ units. The variance is $0.040^2[1+1.80(2.1973)-(2.1973)^2]\approx0.0002032$ units squared, and the conditional standard deviation is approximately $0.01425$ units. The selection rule must be stated precisely: the two thresholds are not interchangeable.

Replication as an Inference Anchor

For an unselected replication estimate with known standard error, the usual normal interval and test apply under the stated model. Independence ensures that these calculations retain their interpretation even after a discovery estimate is selected. This does not mean one should ignore study design: overlap between cohorts, allele or scale mismatches, replication selection based on the replication statistic, or a different target effect can break the argument.

Worked Example: Comparing Replication-Only Inference With Naive Pooling

Suppose a hypothetical discovery estimate is $0.120$ standardized trait units per allele with $s_D=0.040$, and it was reported after exceeding a two-sided threshold $|\hat{\theta}_D/s_D|>1.96$. An independent replication estimate is $0.050$ units with $s_R=0.050$. Both estimates use the same scale and target effect, and the replication estimate was not selected by its result.

The replication-only statistic is $z_R=0.050/0.050=1.00$, with a two-sided normal p-value of about $0.317$. Its 95% interval is $0.050\pm1.96(0.050)=[-0.048,0.148]$ units. The data do not provide strong evidence against a zero effect at the 5% level, but the interval also includes effects in either direction of appreciable size. This is an imprecise estimate, not proof that the effect is zero.

If one incorrectly treats the selected discovery estimate as an ordinary unselected estimate, inverse-variance pooling gives $$ \hat{\theta}_{\mathrm{pool}} =\frac{0.120/0.040^2+0.050/0.050^2}{1/0.040^2+1/0.050^2} =\frac{75+20}{625+400} =0.09268\ \text{units}. $$ The usual pooled standard error would be $\sqrt{1/1025}=0.03123$ units, producing $z\approx2.967$, a two-sided p-value of about $0.0030$, and a nominal 95% interval $[0.0315,0.1539]$ units.

Those pooled results are not justified by the stated sampling process: the discovery estimate was included because it crossed a threshold. Its ordinary normal likelihood cannot be used as though it had been observed without selection. In contrast, the replication-only interval and test remain valid under the independence and no-replication-selection assumptions. The numerical contrast illustrates why a strong naive pooled result does not supersede an inconclusive independent replication.

Combining Estimates With a Known Selection Rule

Discovery information can be incorporated by conditioning its likelihood on the event that caused it to be reported. For the one-sided rule $\hat{\theta}_D/s_D>c$, the conditional discovery density is its normal density divided by the probability of selection under the proposed $\theta$. Multiplication by the independent replication density gives the conditional likelihood:

L(\theta\mid S)\ \propto\ \frac{\phi\!\left((d-\theta)/s_D\right)} {s_D\left[1-\Phi(c-\theta/s_D)\right]} \cdot \frac{1}{s_R}\phi\!\left((r-\theta)/s_R\right),

where $d$ and $r$ are the observed discovery and replication estimates. The denominator matters because the probability of passing the selection rule depends on $\theta$. Omitting it returns the ordinary, selection-naive likelihood. A likelihood interval or test based on this conditional likelihood accounts for the specified selection event; it is not the same as applying the ordinary pooled standard error.

For example, the conditional log-likelihood score (its derivative with respect to $\theta$) is $$ \frac{d-\theta}{s_D^2} -\frac{\lambda(c-\theta/s_D)}{s_D} +\frac{r-\theta}{s_R^2}. $$ The second term subtracts the selection-related contribution that ordinary pooling ignores. A conditional maximum likelihood estimate solves this score equation. Uncertainty can be summarized using a profile likelihood or, as a local approximation, the curvature at the estimate. Neither the variance of selected discovery estimates nor the naive pooled standard error can be substituted for this selection-adjusted uncertainty.

Worked Example: Conditional Likelihood for the Selected Discovery

Use $d=0.120$, $s_D=0.040$, $r=0.050$, and $s_R=0.050$ from the previous hypothetical data, now treating selection as the one-sided rule $d/s_D>c=1.96$. The observed discovery statistic is $0.120/0.040=3.00$, so it passes this rule. The conditional likelihood above combines that selected discovery with the independent replication.

Substitute the values into the score: $$ U(\theta)=\frac{0.120-\theta}{0.0016} -\frac{\lambda(1.96-\theta/0.040)}{0.040} +\frac{0.050-\theta}{0.0025}. $$ Numerically solving $U(\theta)=0$ gives a conditional maximum likelihood estimate of approximately $0.0697$ units. At this value, $a=1.96-0.0697/0.040\approx0.2175$, and $\lambda(a)\approx0.9413$. The discovery score contribution is approximately $31.44-23.54=7.90$; the replication contribution is approximately $(0.050-0.0697)/0.0025=-7.88$. Their near cancellation verifies the numerical solution. The estimate is lower than the naive pooled value $0.09268$ because the conditional likelihood accounts for the discovery threshold.

For a local uncertainty calculation, $\lambda'(a)=\lambda(a)[\lambda(a)-a]$, so the observed information at the estimate is approximately $$ I(\theta)=\frac{1-\lambda'(a)}{s_D^2}+\frac{1}{s_R^2} \approx\frac{1-0.681}{0.0016}+400 \approx599. $$ The corresponding curvature-based standard error is $1/\sqrt{599}\approx0.0409$ units, larger than the naive pooled standard error of $0.03123$. A rough Wald interval is $0.0697\pm1.96(0.0409)=[-0.0105,0.1499]$ units. This interval is an approximation from local likelihood curvature; a profile-likelihood interval is preferable when the likelihood is asymmetric or the approximation is doubtful.

Diagnostics, Alternatives, and Limitations

Before combining estimates, verify that the selection event is known and correctly represented. Check the discovery $Z$-statistic against the actual reporting threshold, confirm that the replication cohort shares no participants with discovery, and harmonize effect alleles and units. Inspect whether the replication was analyzed regardless of outcome or whether only significant replications were retained. If replication results were also selected, its ordinary likelihood is not an unbiased anchor.

Approach	When it is defensible	Main limitation
Replication-only inference	Replication is independent, targets the same effect, and is not selected on its result.	It may be imprecise and does not use discovery information.
Ordinary inverse-variance pooling	Both estimates have their stated unselected sampling distributions and are independent under the fixed-effect model.	It does not account for discovery selection; its usual uncertainty can be misleading.
Conditional-likelihood combination	The sampling model and selection rule are specified, and cohort independence and target-effect assumptions hold.	Misspecified selection or heterogeneity can invalidate its adjustment; likelihood uncertainty may require careful profiling.

A common mistake is to reason, “The discovery estimate has a small standard error, so its large inverse-variance weight makes the pooled result more reliable.” That weight reflects precision under the unselected model; it does not repair selection bias. Another mistake is to use the conditional variance of the truncated discovery estimate as its standard error for $\theta$. The conditional variance describes variation of estimates after selection at a fixed $\theta$; inference about $\theta$ requires a likelihood that includes the selection probability.

The simple conditional model has real limits. A two-sided rule, choice of the most significant locus among many candidates, or selection depending on several correlated tests requires conditioning on the actual selection procedure, not merely inserting a convenient one-sided threshold. If effects vary across cohorts, the common-effect model may also be inappropriate; random-effects methods address heterogeneity, but the selection mechanism must still be represented. When the discovery selection rule or analysis history cannot be reconstructed, replication-only inference is often the clearer basis for a confirmatory conclusion.

Key takeaway: Independence protects an unselected replication estimate from selection of the discovery result. Use replication-only inference as an anchor when its assumptions hold; combine discovery information only with a likelihood or procedure that represents how discovery was selected.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why does an independent replication estimate remain unbiased after conditioning on a discovery threshold, and name one condition that would make this argument fail.
A discovery estimate has $d=0.080$ units and $s_D=0.040$ units. Under the rule $d/s_D>1.96$, did it pass selection?
In the conditional discovery likelihood, what role does the factor $1-\Phi(c-\theta/s_D)$ play?
Why is the variance of a threshold-truncated discovery estimate not automatically the standard error for estimating $\theta$?

Solutions. (1) Independence gives $P(\hat{\theta}_R\mid S)=P(\hat{\theta}_R)$, so its expectation remains $\theta$. The argument fails, for example, if replication results are themselves selected for significance or if cohorts overlap. (2) Its statistic is $0.080/0.040=2.00$, which exceeds $1.96$, so it passes. (3) The factor is the probability of selection under the proposed $\theta$; dividing by it conditions the discovery density on having passed the threshold. (4) That variance describes the spread of selected estimates at a fixed parameter value. Estimation uncertainty must account for both the observed estimate and the probability of selection, which the conditional likelihood does.