Uncertainty in In Silico Functional Prediction

What an In Silico Prediction Does—and Does Not—Say

Computational tools can help prioritize variants when direct experimental evidence is unavailable or incomplete. They use information such as sequence conservation, protein properties, or other annotations to produce a score intended to predict some consequence. But a score is not itself an observation of molecular function. Its meaning depends on the outcome the tool was designed to predict, the data used to train or validate it, and how well it performs for variants like the ones being assessed.

This tutorial focuses on evaluating that uncertainty. The central questions are whether predicted probabilities match observed outcomes (calibration), whether different tools give similar results (agreement), and whether validation evidence applies to the current setting. These questions are related but distinct: agreement between tools does not establish accuracy, and accuracy on one reference set does not guarantee accuracy in another.

A tool may instead output a raw score, such as a value from 0 to 1, without claiming that the value is a probability. A score that ranks variants from lower to higher predicted impact may be useful for prioritization, but interpreting a score of 0.8 as an 80% probability requires evidence that the score has been calibrated that way. The direction of a score also needs to be checked: some tools use larger values for greater predicted impact, while others use smaller values.

“Functional impact” must also be made specific. A prediction may target an effect on protein structure, a particular molecular assay, or a broader label such as a known disease-associated classification. Those are not interchangeable outcomes. A variant’s effect can differ by transcript, tissue, assay, or biological context; a predictor validated for one target should not automatically be read as a probability of every other target.

Assessing Calibration

Calibration is evaluated using variants with both predictions and an independently established reference outcome. For a group of variants assigned similar predicted probabilities, compare the mean predicted probability with the observed fraction whose reference outcome is \(Y=1\). If the mean prediction is much higher than the observed fraction, the tool overpredicts the outcome in that group; if lower, it underpredicts it.

A calibration plot displays observed outcome frequency against mean predicted probability across groups, ideally with points near the 45-degree line. Grouping makes a plot readable but loses detail, and estimates in small groups are uncertain. Calibration can also be summarized by comparing predicted and observed probabilities across the full validation set. Neither a close overall average nor a visually good plot guarantees good calibration in every score range or subgroup.

Calibration requires probability predictions. A ranking score can be evaluated for its ability to distinguish outcomes—for example, whether variants with \(Y=1\) tend to receive higher scores—but ranking performance alone does not show that numerical scores correspond to outcome probabilities. Converting a score into probabilities requires a calibration procedure fitted on appropriate data and checked on data not used to fit that conversion.

One overall measure for probabilistic predictions is the Brier score, the mean squared difference between each predicted probability and its binary reference outcome. For \(m\) validated variants,

A lower value indicates smaller squared probability errors on that validation set. The Brier score reflects both calibration and how well predictions distinguish outcomes; it does not identify which aspect accounts for a particular value. It also depends on the outcome frequency in the evaluation set, so comparisons should use relevant, comparable validation data.

Agreement Between Prediction Tools

Agreement asks whether tools give similar predictions for the same variants. Before measuring it, specify what counts as a matching result. For example, two tools may be reduced to high- and low-impact categories using stated cutoffs, or their numerical scores may be compared on a common scale. Different cutoffs can produce different agreement, and a high correlation between numerical scores does not necessarily mean that the tools assign similar values: one tool could be a shifted or rescaled version of the other.

For binary classifications, the proportion of variants with matching calls is a direct measure of observed agreement. It does not account for the amount of agreement expected from the tools’ marginal call frequencies. Cohen’s kappa makes that comparison: if \(P_o\) is observed agreement and \(P_e\) is the agreement expected from the two tools’ marginal proportions under independence, then \(\kappa=(P_o-P_e)/(1-P_e)\), provided \(P_e\ne1\). Kappa is a descriptive summary, not a universal measure of biological validity; it can change with the mix of variants and the prevalence of each category.

Validation Metrics and the Importance of the Reference Set

If a tool is used to classify variants as positive or negative for a specified reference outcome, its performance can be summarized by sensitivity and specificity. Sensitivity is the proportion of reference-positive variants classified positive; specificity is the proportion of reference-negative variants classified negative. These quantities depend on the chosen threshold and on how the reference outcome is defined.

The positive predictive value (PPV), the proportion of tool-positive variants that are reference-positive, depends not only on sensitivity and specificity but also on the prevalence of the reference outcome in the population being assessed. This dependence follows from the law of total probability:

where \(T=1\) denotes a positive tool classification. In this expression, \(P(T=1\mid Y=1)\) is sensitivity, \(P(T=1\mid Y=0)=1-\text{specificity}\), and \(P(Y=1)\) is the outcome prevalence. The denominator adds the probabilities of a positive tool result arising from a reference-positive variant and from a reference-negative variant. Consequently, PPV from an enriched benchmark set may not transfer to a setting where the outcome is less common.

Limits of Computational Evidence

A validation result is only as meaningful as its reference labels and sampling design. Experimental assays can provide direct measurements of a specified molecular property, but an assay is not a universal test of all biological function. Results may depend on the experimental system, construct, cell type, conditions, and measured endpoint. Conversely, labels based on previous classifications or expert curation may incorporate evidence other than direct functional experiments. It is important to identify what the reference outcome actually represents rather than treating every benchmark label as experimentally established function.

Performance can also vary across variant classes, genes, transcripts, and populations. A tool validated mainly on well-studied variants may not be equally reliable for rare variants with less direct evidence. A change in transcript choice or target assay can change the relevant interpretation. Independent validation, appropriate reference outcomes, and assessment in a population resembling the intended use all strengthen an evaluation, but none turns a prediction into a direct experimental observation.

Common Mistake

Wrong reasoning: “Two prediction tools both assign a high-impact score, so the variant’s damaging function has been experimentally confirmed.” This conflates agreement with validation and prediction with measurement. If tools share information or model assumptions, they may make the same error; even independent agreement would still be computational evidence rather than an experimental result.

Correction: Report the tools’ outputs as predictions, identify the specific outcome they target, and assess their reliability using relevant validation evidence. State separately whether direct experimental evidence exists and what that experiment measured. A high score can prioritize follow-up or contribute to an evidence assessment, but by itself it does not establish a molecular effect, a clinical consequence, or causation.