Evaluating Replication Across Ancestry Groups

What Counts as Replication Across Ancestry Groups?

An association found in one ancestry group may not reach a conventional significance threshold in another. That difference alone does not establish that the association failed to replicate: the groups may have different sample sizes, allele frequencies, or standard errors, even when their underlying effects are similar. Conversely, two significant results do not by themselves show that the effects are equal or that one result has independently replicated the other.

A careful assessment uses the estimated effects and their uncertainty, asks whether the estimates point in a consistent direction, and evaluates evidence for heterogeneity. It also considers how much power each group had to detect an effect of the relevant size. As established in the earlier material on allele harmonization and effect scales, alleles, effect directions, phenotypes, and scales must be aligned before estimates can be compared.

Replication is best treated as an evidence assessment, not a binary label produced by one p-value. A replication analysis should specify in advance which group provides the discovery evidence, which group is the replication sample, the expected direction, the effect scale, and the statistical criterion. For a planned one-sided test, the direction must be prespecified; if direction was selected after seeing the replication data, a two-sided assessment is generally appropriate.

Estimates, Intervals, and Direction

Suppose \(\hat{\beta}_k\) is the estimated association effect in ancestry group \(k\), with standard error \(SE_k\). The estimate describes the direction and magnitude on the stated scale; its standard error describes sampling uncertainty under the fitted model. Under an approximate normal sampling distribution, a two-sided 95% confidence interval is \(\hat{\beta}_k \pm 1.96SE_k\). This interval gives a range of effect values compatible with the data and model at that confidence level; it is not a 95% probability statement about the fixed effect after the interval is observed.

The sign of \(\hat{\beta}_k\) is informative only when effect alleles and outcome coding are aligned. A positive estimate in one analysis and a negative estimate in another may reflect allele flips or coding differences rather than biological disagreement. Even with harmonization, a small estimate with a wide interval can have either sign because the data are imprecise. Thus, distinguish the point-estimate direction from the strength of evidence about direction.

Comparing p-values or significance labels discards important information. A p-value depends on both the estimated effect and its standard error. A smaller study can estimate an effect in the same direction but have a larger p-value because its standard error is larger. For this reason, report the effect estimate, interval, and sample-specific evidence together.

Heterogeneity: Is There Evidence That Effects Differ?

Directional replication asks whether estimates point in the same direction. Heterogeneity analysis asks a different question: whether the effect estimates are more dispersed than would be expected from sampling error under a common-effect model. For independent estimates \(\hat{\beta}_1,\ldots,\hat{\beta}_K\) with standard errors \(SE_k\), an inverse-variance common-effect estimate is

The Cochran heterogeneity statistic is \(Q=\sum_k w_k(\hat{\beta}_k-\hat{\beta}_{FE})^2\). Under independent, approximately normal estimates with a common true effect and valid standard errors, \(Q\) is approximately chi-square distributed with \(K-1\) degrees of freedom. A small p-value is evidence against the common-effect assumption. A large p-value does not prove equal effects: with few groups or imprecise estimates, the test may have low power to detect heterogeneity.

The common-effect calculation assumes that the estimates target comparable parameters and that their sampling errors are adequately characterized. Dependence between estimates, overlapping participants, or unmodeled uncertainty can invalidate the usual reference distribution unless accounted for. The heterogeneity test should therefore be interpreted alongside the estimates and intervals, not as a stand-alone verdict.

Do not use interval overlap as a substitute for a heterogeneity test. Overlap is not the test of the difference between estimates; the uncertainty of a difference depends on the standard errors and their covariance. With independent estimates, the standard error of \(\hat{\beta}_1-\hat{\beta}_2\) is \(\sqrt{SE_1^2+SE_2^2}\). If samples overlap, covariance must also be considered.

Ancestry-Specific Power and Interpretation

A nonsignificant result is more informative when a study had substantial power to detect an effect of scientific interest. Under a normal approximation, if the true effect is \(\beta\) and its standard error is \(SE\), the test statistic has approximate distribution \(N(\delta,1)\), where \(\delta=\beta/SE\). For a two-sided test at level \(\alpha\), approximate power is

This calculation assumes a prespecified effect size, valid standard error, correct model, and the specified testing threshold. In ancestry-stratified analyses, power can differ because sample sizes and allele frequencies differ, among other reasons. Differences in power do not themselves explain a particular observed effect difference, but they help calibrate how much weight to place on a nonsignificant result.

Diagnostics, Alternatives, and Limitations

Before drawing a replication conclusion, verify that the comparisons are meaningful. Check allele alignment and strand handling, outcome coding, effect scale, phenotype definition, covariate adjustment, and analysis model. Review ancestry-specific sample sizes, allele counts or frequency, missingness, imputation quality, and calibration diagnostics. A reversed effect allele can create an apparent direction reversal; sparse data or poor calibration can make an estimate unstable. Large discovery estimates can also be inflated by selection, so a smaller independent estimate is not automatically evidence of biological nonreplication.

The main approaches answer related but distinct questions. A direction check is simple and useful for assessing sign consistency, but it ignores precision and magnitude. Comparing estimates and confidence intervals makes both visible, but interval overlap is not a formal equality test. A heterogeneity test evaluates dispersion beyond sampling error, but can have low power and depends on its assumptions. An ancestry-specific power calculation contextualizes a null result, but depends on the assumed effect and does not prove that low power caused the result.

These tools also have limits. Statistical compatibility across groups does not establish a shared causal mechanism, and apparent heterogeneity can arise from differences in tagging, measurement, or model specification as well as effect variation. A p-value above a threshold is not evidence that the null is true; a p-value below it is not a measure of effect size. Report estimates and uncertainty in context, and avoid treating socially defined ancestry categories as homogeneous biological populations.

A practical conclusion should be calibrated: for example, “The two estimates have the same positive direction; the replication estimate is less precise and its interval includes zero, while the heterogeneity test provides limited evidence of a difference.” Such wording communicates what the data show without claiming either exact equality or definitive failure.