Tutorials › Statistical Genetics › Uncertainty and Sample Size in Portability Gaps

Home  /  Tutorials  /  Ancestry Portability of Scores

Ancestry Portability of Scores · Tutorial 750 of 989

Uncertainty and Sample Size in Portability Gaps

Estimate and interpret uncertainty in portability gaps, then assess what sample sizes can support precise comparisons or meaningful claims of equivalence.

Advanced 12 min read

What You'll Learn

  • Define an ancestry portability gap with an explicit direction and performance scale.
  • Construct confidence intervals using standard errors that reflect unequal sample sizes.
  • Account for covariance when comparing correlated performance-gap estimates.
  • Use precision targets and pilot standard errors to plan validation sample sizes.
  • Distinguish failure to detect a gap from evidence that a gap is small enough to be equivalent.

Why Uncertainty Matters for Portability Gaps

A difference in estimated score performance across ancestry groups is not automatically a reliable difference in the populations. Validation samples are finite, and performance estimates vary from one sample to another. That variation can be substantial when one ancestry group has far fewer participants than another. A point estimate alone therefore cannot show whether a gap is precisely estimated, compatible with no gap, or small enough to meet a prespecified standard for practical equivalence.

The target of inference must also be clear. Comparisons should use the same performance metric and a comparable outcome scale in each group, as established in earlier material on comparing score performance across outcome scales. Ancestry groups and the validation design should be defined before interpreting the results. A confidence interval quantifies sampling uncertainty under the analysis and sampling assumptions; it does not correct a poorly chosen target population, an invalid validation design, or systematic measurement differences.

Definition: Let \(\theta_A\) and \(\theta_B\) denote the population values of the same performance metric for two defined ancestry groups, A and B. When larger values indicate better performance, define the portability gap as \(\Delta=\theta_A-\theta_B\). A positive gap means the metric is higher in group A; a negative gap means it is higher in group B. If a metric is better when smaller, reverse the subtraction or otherwise define the direction explicitly. The gap is expressed in the units of that metric.

Confidence Intervals for a Gap

Let \(\hat{\theta}_A\) and \(\hat{\theta}_B\) be estimates of the group-specific performance values. The estimated gap is \(\hat{\Delta}=\hat{\theta}_A-\hat{\theta}_B\). If the estimates are approximately unbiased and their sampling distributions are sufficiently well approximated by a normal distribution, an approximate confidence interval is

$$ \hat{\Delta}\ \pm\ z_{1-\alpha/2}SE(\hat{\Delta}). $$

Here \(z_{1-\alpha/2}\) is the standard normal quantile for the chosen confidence level; for a 95% interval, it is approximately 1.96. A confidence interval procedure with 95% coverage would include the true gap in 95% of repeated samples under its assumptions. It does not assign a 95% probability to the fixed true gap being inside the particular interval calculated from these data.

If the ancestry groups consist of independent participants, the two estimates have zero sampling covariance, so

$$ \operatorname{Var}(\hat{\Delta}) =\operatorname{Var}(\hat{\theta}_A)+\operatorname{Var}(\hat{\theta}_B), \qquad SE(\hat{\Delta})= \sqrt{SE(\hat{\theta}_A)^2+SE(\hat{\theta}_B)^2}. $$

The standard errors need not be equal. In particular, a smaller validation sample often contributes more uncertainty, though the contribution also depends on the metric and the data. For an approximately sample-mean-like estimator, a common large-sample form is

$$ \operatorname{Var}(\hat{\Delta})\ \approx\ \frac{v_A}{n_A}+\frac{v_B}{n_B}, $$

where \(n_A\) and \(n_B\) are the group sample sizes, and \(v_A\) and \(v_B\) are per-participant variance contributions for the estimator. For a score-performance metric such as \(R^2\), those contributions are not generally the variance of a single raw outcome; use a valid metric-specific standard error, often obtained by resampling participants. If the same participants contribute to both group estimates, the covariance must also be included: subtract twice the covariance when calculating the variance of their difference.

Worked Example: An Unequal-Sample-Size Gap in R²

Suppose a hypothetical validation study estimates \(R^2\) as 0.145 in group A, based on \(n_A=800\), and 0.090 in group B, based on \(n_B=250\). Assume participant-level bootstrap standard errors of 0.018 and 0.030, respectively. The validation groups contain different participants, and the bootstrap procedure is appropriate for these metric estimates.

Estimate and direction. With group A minus group B as the defined direction,

$$ \hat{\Delta}=0.145-0.090=0.055. $$

Standard error. Because the groups are independent, add the estimated variances:

$$ SE(\hat{\Delta})= \sqrt{0.018^2+0.030^2} =\sqrt{0.000324+0.000900} =\sqrt{0.001224} \approx 0.0350. $$

95% confidence interval. Using the normal approximation,

$$ 0.055\pm1.96(0.0350) =0.055\pm0.0686 =(-0.0136,\ 0.1236). $$

The estimated \(R^2\) gap is 0.055, or 5.5 percentage points of \(R^2\). The interval ranges from a small advantage for group B to a larger advantage for group A and includes zero. These data do not precisely determine the direction or size of the population gap. They do not establish that the groups have equal performance. The larger standard error in group B contributes substantially to the uncertainty, despite group A's larger sample.

Correlated Performance Estimates

Performance measures computed on the same participants can be correlated. The same is true of gap estimates for related outcomes evaluated in the same validation sample. This matters when the scientific question concerns a contrast between those gaps. Correlation does not mean that all analyses need the same adjustment: it is the covariance of the particular estimates entering a contrast that determines its variance.

For two gap estimates, \(\hat{\Delta}_1\) and \(\hat{\Delta}_2\), the variance of their difference is

$$ \operatorname{Var}(\hat{\Delta}_1-\hat{\Delta}_2) =\operatorname{Var}(\hat{\Delta}_1)+\operatorname{Var}(\hat{\Delta}_2) -2\operatorname{Cov}(\hat{\Delta}_1,\hat{\Delta}_2). $$

This contrast is meaningful only when the gaps are on a common scale and address a sensible comparison—for example, \(R^2\) gaps for two related traits analyzed comparably. The covariance can be estimated by resampling whole participants, preserving the measurements for both traits together within each ancestry group. Resampling each trait independently would discard their observed dependence and can give the wrong standard error for the contrast.

Worked Example: Comparing Correlated Gaps Across Two Traits

In a hypothetical study, two related traits are measured for participants in each of two ancestry groups. For each trait, the gap is the group A minus group B difference in \(R^2\). A paired participant bootstrap gives \(\hat{\Delta}_1=0.050\), \(SE(\hat{\Delta}_1)=0.020\), \(\hat{\Delta}_2=0.030\), and \(SE(\hat{\Delta}_2)=0.018\). The estimated correlation between the two gap estimates is 0.70. Assume the traits use comparable \(R^2\) definitions and the bootstrap preserves the paired participant records.

The estimated contrast is \(0.050-0.030=0.020\). First convert the correlation to covariance:

$$ \operatorname{Cov}(\hat{\Delta}_1,\hat{\Delta}_2) =0.70(0.020)(0.018)=0.000252. $$

Then calculate the contrast variance and standard error:

$$ \begin{aligned} \operatorname{Var}(\hat{\Delta}_1-\hat{\Delta}_2) &=0.020^2+0.018^2-2(0.000252)\\ &=0.000400+0.000324-0.000504\\ &=0.000220,\\ SE(\hat{\Delta}_1-\hat{\Delta}_2)&=\sqrt{0.000220}\approx0.0148. \end{aligned} $$

The approximate 95% interval for the difference between the gaps is

$$ 0.020\pm1.96(0.0148) \approx(-0.0091,\ 0.0491). $$

The estimated gap for trait 1 exceeds the gap for trait 2 by 0.020 \(R^2\) units, but the interval includes zero. For comparison, incorrectly treating the gap estimates as independent would give \(SE=\sqrt{0.000400+0.000324}=0.0269\), and a wider interval, approximately \((-0.0327,\ 0.0727)\). Here positive covariance reduces the variance of a difference; covariance can have the opposite effect for other contrasts or signs.

Planning Sample Size for Useful Precision

A sample-size calculation should begin with the intended precision or decision, not simply a desire to recruit equal numbers. For a gap with independent group estimates, plan using the anticipated variance contributions from both groups. If the approximate standard error is \(SE(\hat{\Delta})\), a normal-approximation confidence interval has half-width \(z_{1-\alpha/2}SE(\hat{\Delta})\). Requiring a half-width no larger than \(h\) therefore requires

$$ SE(\hat{\Delta})\leq\frac{h}{z_{1-\alpha/2}}. $$

This is a precision criterion, not a guarantee of detecting a nonzero gap. Calculations based on pilot standard errors also assume that the metric, validation procedure, participant mix, and relevant variability will be sufficiently similar in the planned study. If those assumptions are uncertain, evaluate a range of plausible variance contributions rather than treating one estimate as exact.

Worked Example: Planning the Smaller Ancestry Group

Suppose a hypothetical pilot gives standard errors for \(R^2\) estimates of 0.020 with 400 participants in group A and 0.030 with 225 participants in group B. For planning only, assume each standard error scales with the inverse square root of its group sample size. A proposed study can enroll \(n_A=900\), and the goal is a 95% interval for the gap with half-width at most 0.040 \(R^2\) units. Assume independent groups and comparable sampling and analysis procedures.

First project the group A standard error:

$$ SE_A=0.020\sqrt{\frac{400}{900}}\approx0.01333. $$

A 95% half-width of 0.040 requires a gap standard error of at most

$$ \frac{0.040}{1.96}\approx0.02041. $$

The group B standard error must therefore satisfy

$$ SE_B\leq \sqrt{0.02041^2-0.01333^2} \approx0.01545. $$

Use the pilot scaling assumption to solve for \(n_B\):

$$ 0.030\sqrt{\frac{225}{n_B}}\leq0.01545 \quad\Longrightarrow\quad n_B\geq225\left(\frac{0.030}{0.01545}\right)^2 \approx848.3. $$

Thus, plan for at least 849 participants in group B under these assumptions; rounding up to 850 allows a small practical buffer. The calculation shows why the smaller group may need substantial recruitment: its uncertainty otherwise dominates the gap. It does not account for attrition, exclusions, or departures from the pilot assumptions, which should be considered in an actual design.

With a fixed total sample size and equal recruitment costs, approximate variance contributions \(v_A/n_A+v_B/n_B\) are minimized by allocating participants in proportion to the group-specific standard deviations, not necessarily by splitting the sample equally. To see this, minimize \(v_A/n_A+v_B/n_B\) subject to \(n_A+n_B=N\). Differentiating after setting \(n_B=N-n_A\) gives

$$ -\frac{v_A}{n_A^2}+\frac{v_B}{n_B^2}=0 \quad\Longrightarrow\quad \frac{n_A}{n_B}=\sqrt{\frac{v_A}{v_B}}. $$

This allocation is a useful benchmark, not a universal rule. Recruitment costs, the goal of estimating each group’s performance separately, and the target populations may justify a different allocation. A design intended to establish equivalence also needs enough precision relative to a prespecified equivalence margin.

Imprecision Is Not Equivalence

A confidence interval that includes zero does not establish that the groups have the same performance. It may include zero because the estimated gap is near zero, because the study is imprecise, or both. Equivalence asks a different question: is the gap sufficiently small, in either direction, to fall within a prespecified range that is negligible for the scientific or practical purpose?

Let \(\delta>0\) be a prespecified margin, expressed in the units of the gap. Equivalence means \(-\delta<\Delta<\delta\). At significance level \(\alpha=0.05\), the usual two one-sided tests approach declares equivalence when the 90% confidence interval lies entirely inside \((-\delta,\delta)\). The margin must be justified in context; it should not be chosen after looking at the observed gap or its interval.

Worked Example: Evidence of Equivalence Within a Prespecified Margin

Suppose a hypothetical validation study estimates an \(R^2\) gap of \(\hat{\Delta}=0.004\), group A minus group B, with \(SE(\hat{\Delta})=0.010\). Before examining the results, investigators specified that gaps between \(-0.030\) and \(0.030\) \(R^2\) units are negligible for their purpose. Assume the standard error and normal approximation are adequate.

The 90% interval uses \(z_{0.95}\approx1.645\):

$$ 0.004\pm1.645(0.010) =0.004\pm0.01645 =(-0.01245,\ 0.02045). $$

The entire interval falls within the prespecified equivalence range \((-0.030,\ 0.030)\). Under the stated procedure and assumptions, this supports equivalence at the 5% level. The conclusion is not that the true gap is exactly zero; it is that the data support a gap small enough to lie within the chosen margin. If the same estimate had a much wider interval extending outside the margin, a nonsignificant test of zero would not be evidence of equivalence.

Common Mistakes

  • Using the larger group’s standard error for the gap. That ignores the other group’s uncertainty. For independent estimates, combine both variances, including the larger contribution from a smaller or more variable group.
  • Ignoring dependence between estimates. When a contrast involves correlated estimates, adding variances as if the estimates were independent is generally wrong. Estimate the covariance using a resampling or model-based procedure that respects the data structure.
  • Calling a nonsignificant result “equal performance.” A test that does not reject a zero gap may reflect low precision. Evidence of equivalence requires a prespecified margin and an interval sufficiently narrow to fall inside it.
  • Interpreting an interval as a probability statement about the fixed gap. A confidence interval describes the long-run coverage of a procedure under its assumptions, not the probability that the realized interval contains the parameter.
  • Comparing gaps without checking scale and direction. Define which group is subtracted from which, and use a common, interpretable metric before comparing estimates.
Key takeaway: Report the direction and scale of a portability gap, its standard error and confidence interval, and the group-specific sample sizes. When estimates are correlated, account for their covariance. A wide interval signals uncertainty; equivalence requires a narrow interval contained within a scientifically justified margin.

Check Your Understanding

Try these questions without looking back.

  1. If group A has estimated \(R^2=0.12\) and group B has estimated \(R^2=0.08\), what is the gap under the convention \(\Delta=\theta_A-\theta_B\), and what does its sign mean?
  2. Two independent ancestry-specific performance estimates have standard errors 0.02 and 0.03. Write the formula for the standard error of their difference and calculate it.
  3. Why should a bootstrap preserve paired participant records when estimating the covariance between two performance gaps for related traits?
  4. A 95% confidence interval for a gap includes zero but extends beyond a prespecified equivalence margin. What conclusion is justified, and what conclusion is not?
  5. Under inverse-square-root sample-size scaling, what happens to a group’s standard error when its sample size is multiplied by four?

Previous

Next

Back to Tutorials
Clinical Biostats