Why Validate by Masking?
Imputation produces predictions for genotypes that were not directly observed. To assess how well those predictions perform, a validation study can temporarily hide some genotypes that are known, impute them as though they were missing, and compare the predictions with the withheld values. The known genotypes serve as a reference for evaluation; they are not available to the imputation procedure for those tests.
The comparison is meaningful only if the masking experiment resembles the intended use. A study that masks a convenient set of easy-to-impute genotypes may report high accuracy while saying little about the genotypes that will actually be imputed. In particular, performance can differ across allele frequencies, individuals, variants, and data-generation or quality-control conditions.
Designing a Credible Masking Study
Begin by defining the evaluation units: the individual–variant genotype pairs that will be hidden and scored. Select them from genotypes with sufficiently reliable observed values to serve as truth. If truth calls are uncertain or systematically erroneous, the validation measures agreement with those calls, not necessarily agreement with the unobserved biological genotypes.
The masking step must occur before any step that could use a test genotype. For example, if haplotypes or phase are estimated using the target genotype, hiding it only after that estimation leaves information about the answer in the input. Re-run relevant preprocessing and imputation with the masked calls unavailable. Keep the reference information and other pipeline choices consistent with the intended analysis, and do not use the held-out values to choose settings and then report performance on those same values as an independent evaluation.
A useful design samples masked genotypes across both individuals and variants, rather than repeatedly testing only a small set of subjects or loci. Mask selection should be independent of the unknown prediction errors. If the actual missingness pattern is known to depend on features such as assay or sample quality, a completely random mask may not reproduce it; an additional validation analysis can mimic that pattern. Record the sampling design, because it determines what population of genotypes the resulting accuracy estimates describe.
Specify the genotype pairs and data setting for which imputation performance is to be assessed.
Use observed genotypes that are credible evaluation targets, and document exclusions or uncertainty.
Remove each selected genotype before any phasing or imputation operation that could use it.
Save dosages and, if needed, hard calls; compare each prediction with its withheld truth.
Summarize accuracy within allele-frequency strata and state how test pairs were sampled and weighted.
Hard-Call Accuracy Measures
A hard-call evaluation requires a pre-specified rule for converting imputation output into one genotype per test pair. If genotype posterior probabilities are available, a common rule selects the genotype with the largest posterior probability. This rule can produce a hard call even when the prediction is uncertain, so report the rule and do not confuse a discrete call with the full uncertainty in the imputation output.
Here \(\mathcal{M}\) is the set of masked pairs, \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise, and \(N\) is the number of pairs in that set. Concordance can look high when most truth genotypes are reference homozygotes, even if many non-reference genotypes are missed. Sensitivity and positive predictive value help reveal different errors: missed non-reference genotypes and incorrect non-reference predictions, respectively.
Worked Example: Reading a Hard-Call Confusion Table
Suppose a hypothetical validation set contains 150 masked pairs at one variant. Rows below are truth genotypes and columns are predicted hard calls.
| Truth \ Predicted | 0 | 1 | 2 | Row total |
|---|---|---|---|---|
| 0 | 80 | 8 | 2 | 90 |
| 1 | 6 | 24 | 5 | 35 |
| 2 | 1 | 4 | 20 | 25 |
The diagonal contains exact matches, so concordance is \((80+24+20)/150=124/150=0.8267\), or 82.67%. There are \(35+25=60\) truth non-reference genotypes. Of these, \(24+5+4+20=53\) are predicted non-reference, giving sensitivity \(53/60=0.8833\), or 88.33%.
There are \(8+24+4=36\) predicted heterozygotes and \(2+5+20=27\) predicted alternate homozygotes, so 63 calls are non-reference. Positive predictive value is \(53/63=0.8413\), or 84.13%. The three values answer different questions; none alone describes every type of error.
Dosage Accuracy and Frequency-Stratified Reporting
A dosage is an expected alternate-allele count and can take a fractional value between 0 and 2. It retains information that a hard call discards. For test pairs, dosage errors can be summarized in allele-count units: mean absolute error measures typical absolute discrepancy, root mean squared error gives greater weight to large discrepancies, and mean error shows the direction of average over- or underprediction.
Each of these measures is computed over a stated set of masked pairs. MAE and RMSE are nonnegative; mean error can be positive or negative. Their units are alternate-allele copies per genotype. A near-zero mean error does not imply small individual errors, because positive and negative errors can cancel.
Report accuracy by allele-frequency strata because a pooled result can conceal important differences. For a biallelic variant, the minor-allele frequency is the smaller of the two allele frequencies. For example, if \(A_j\) alternate alleles are observed among \(N_j\) truth genotypes at variant \(j\), the alternate-allele frequency is \(A_j/(2N_j)\), and the minor-allele frequency is the smaller of that value and its complement. Define the frequency bins before examining prediction errors.
For evaluation, a bin can be assigned using allele frequency calculated from the full set of reliable truth genotypes at a variant, including masked truth, because that calculation is used only to describe and group the evaluation results. It must not be fed into the imputation prediction as if it were available for a missing genotype. Alternatively, use an external frequency estimate and state its source. Report the bin boundaries and the number of test pairs or variants in each bin.
Worked Example: Calculating Dosage Error
Consider six hypothetical masked genotypes with truth values \(0,0,1,1,2,2\) and imputed dosages \(0.1,0.2,0.7,1.4,1.6,1.9\). All dosages lie between 0 and 2, as required for expected alternate-allele counts.
The prediction-minus-truth errors are \(0.1,0.2,-0.3,0.4,-0.4,-0.1\). The absolute errors sum to \(0.1+0.2+0.3+0.4+0.4+0.1=1.5\), so \(\mathrm{MAE}=1.5/6=0.2500\) allele copies per genotype.
The squared errors sum to \(0.01+0.04+0.09+0.16+0.16+0.01=0.47\). Thus \(\mathrm{RMSE}=\sqrt{0.47/6}=\sqrt{0.07833}=0.2799\) allele copies per genotype. The signed errors sum to \(-0.1\), so mean error is \(-0.1/6=-0.01667\) allele copies per genotype. This small average bias does not mean every prediction is close: MAE and RMSE show the typical discrepancy more directly.
Worked Example: Why Frequency-Stratified Results Matter
Suppose a hypothetical evaluation reports concordance for three frequency strata. The low-frequency stratum has 2,000 test pairs and concordance 0.90; the intermediate stratum has 5,000 pairs and concordance 0.96; the common stratum has 8,000 pairs and concordance 0.99.
The pooled concordance weights each pair equally: \((2{,}000(0.90)+5{,}000(0.96)+8{,}000(0.99))/15{,}000 =(1{,}800+4{,}800+7{,}920)/15{,}000=0.9680\). The equally weighted mean of the three stratum values is \((0.90+0.96+0.99)/3=0.9500\).
Both summaries are arithmetically valid, but they answer different questions. The pooled value describes the performance for a test pair drawn from this particular mix of strata; the equal-stratum mean gives each stratum equal weight. Since the common stratum supplies over half of the pairs, the pooled result is closer to its higher concordance and obscures the lower value in the low-frequency stratum. Report the stratum-specific results alongside any overall summary.
Justification, Diagnostics, and Alternatives
The basic concordance estimate has a simple justification. Define \(Y_{ij}=I(\hat{G}^{\,\mathrm{hard}}_{ij}=G_{ij})\), which is 1 for a match and 0 otherwise. The observed concordance is \(\bar{Y}=N^{-1}\sum_{\mathcal{M}}Y_{ij}\). If test pairs are sampled representatively from a target population and the prediction procedure is fixed, the sample mean estimates the target probability of a match: each sampled pair contributes one match indicator, and averaging those indicators estimates their mean. If selection probabilities differ, an unweighted average instead describes the selected test set; representing a different target population requires appropriate design weights.
This argument does not require all genotype pairs to be independent for the observed concordance to be calculated. However, uncertainty calculations that treat every pair as an independent Bernoulli observation can be misleading: genotypes from the same person and nearby or related variants can be dependent. When uncertainty intervals are needed, use a resampling or variance method that respects the sampling structure, such as resampling individuals and, where appropriate, genomic regions. State what was resampled and what population the interval is intended to describe.
Useful diagnostics include the number of masked pairs and variants in every frequency bin, the truth-genotype counts, the predicted-genotype counts, and the full confusion table. Check that masked genotypes were genuinely withheld before preprocessing; that truth calls are reliable; and that the evaluation set is not dominated by one individual, genomic region, or variant. Very small denominators make sensitivity or positive predictive value unstable, so show counts and avoid treating a noisy stratum estimate as precise.
Masked validation is not the only assessment. It is more directly empirical than an imputation quality summary computed from the imputation output, but it requires suitable observed genotypes to hide. An independent set of genotypes measured with a trusted assay can provide a useful validation target and may better represent the intended samples, provided it is processed compatibly and does not leak into model construction. Simulation can provide truth under specified assumptions, but its conclusions depend on how well those assumptions represent real data. These approaches complement one another; none makes the others automatically unnecessary.
Common Mistakes and Limitations
- Using genotypes that were not truly held out. Hiding calls only after phasing or another genotype-dependent step can leak information. Mask first, then rerun affected steps.
- Relying on overall concordance alone. A high value can arise when reference genotypes dominate. Include frequency-stratified results and non-reference measures.
- Confusing sensitivity with positive predictive value. Sensitivity divides by truth non-reference calls; positive predictive value divides by predicted non-reference calls. State the denominator.
- Equating a convenient mask with the actual missingness process. Results from randomly masked, high-confidence calls may not generalize to calls missing because of particular assay or sample failures.
- Using a single metric as a complete quality judgment. Hard calls and dosages answer different questions, and no one accuracy measure captures every relevant error.
Masked-genotype validation estimates performance under its chosen truth set, masking scheme, reference information, and pipeline. Differences between that setup and the intended application limit transportability. Report those conditions with the numerical results rather than presenting accuracy as a property of imputation in general.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why must a genotype be hidden before phasing if phasing uses observed genotypes?
- A test set has 40 truth non-reference genotypes, of which 32 are predicted non-reference. What is non-reference sensitivity?
- Why might pooled concordance exceed the equal-weighted average of frequency-stratum concordances?
- Dosage predictions are unbiased on average but sometimes far from truth. Which error summaries help reveal this?
- What does masked validation estimate if the masked calls are easier to impute than the calls that are actually missing?
Solutions. (1) Otherwise the phasing step can carry information from the nominally hidden genotype into the prediction. (2) Sensitivity is \(32/40=0.8000\), or 80.00%. (3) The pooled summary weights strata according to their numbers of test pairs; if higher-accuracy strata contribute more pairs, they pull the pooled value above an equally weighted mean. (4) MAE and RMSE measure the size of discrepancies; mean error alone can be near zero because positive and negative errors cancel. (5) It estimates performance for the easier masked set, which may overstate performance for the actual missing genotypes.