Why Calibrate an Interaction Score?
An interaction database may attach a confidence score to each candidate relationship. That score can help order candidates from stronger to weaker evidence, but its numeric value does not automatically say how likely an interaction is to be present in the population being modeled. A score of 0.8, for example, is not a prior inclusion probability of 0.8 unless it has been calibrated for a clearly defined target event and population.
The distinction matters when biological knowledge enters a statistical model as a soft prior. A ranking can prioritize candidates, but a probability or prior odds weight specifies how strongly a candidate is favored before the current omics data are analyzed. Calibration is the step that gives the score a probability interpretation; it cannot make an ill-defined target event or unrepresentative validation data reliable.
Before fitting a mapping, specify what “present” means. It might mean a physical interaction supported by a defined assay, or a relationship accepted under a specified evidence standard; these are different events. Also specify the candidate population to which the prior will be applied. A calibration learned among well-studied proteins may not transport to a population containing many understudied candidates.
From Scores to Probabilities and Prior Weights
Suppose a validation set contains \(m\) candidate interactions with scores \(s_i\) and binary reference labels \(y_i\), where \(y_i=1\) means the candidate meets the chosen target definition. A common calibration model is logistic:
Here \(a\) and \(b\) are estimated from labeled validation data. The logistic form keeps fitted probabilities between 0 and 1. A positive \(b\) represents higher fitted inclusion probability at higher scores; the data must support that direction rather than the analyst assuming it. This model assumes that the log-odds are adequately described as linear in the chosen predictor and that the validation labels are suitable for the target population. If these assumptions are doubtful, a more flexible calibration method or a revised validation design is needed.
The score itself may already be a probability, a log-odds value, or an arbitrary ranking score. These are not interchangeable. If a source supplies a probability \(p\), a logistic recalibration model may use \(\operatorname{logit}(p)\) as its predictor. Identity coefficients, \(a=0\) and \(b=1\), reproduce \(p\) only when the predictor is \(\operatorname{logit}(p)\), not when it is \(p\). A score merely ranging from 0 to 1 is not thereby a probability.
Once a calibrated probability \(\pi_j\) is available, its prior inclusion odds are \(\pi_j/(1-\pi_j)\). If a model uses a common baseline inclusion probability \(\pi_0\), a useful relative prior odds weight is the ratio of the candidate’s prior odds to the baseline odds. This weight expresses relative prior preference; it is not itself an inclusion probability.
The conversion follows directly from the definition of odds: the odds of inclusion are the probability of inclusion divided by the probability of non-inclusion. Dividing these odds by the corresponding baseline odds gives a dimensionless multiplier. A value \(w_j>1\) favors inclusion relative to baseline, \(w_j=1\) matches baseline, and \(0<w_j<1\) disfavors inclusion relative to baseline. The model must still use these quantities according to its stated prior specification.
Worked Example: Logistic Calibration of a Raw Score
Suppose a hypothetical, representative validation set was used to fit \(\operatorname{logit}\{\pi(s)\}=-3+5s\), where \(s\) is a raw confidence score, not a probability. For a candidate with score \(s=0.80\), calculate its calibrated prior inclusion probability and its odds weight relative to a baseline probability of \(\pi_0=0.10\).
First calculate the fitted log-odds and transform them to a probability:
The calibrated prior inclusion probability is approximately 0.731, under the stated target definition and calibration model. Its prior odds are \(0.7311/(1-0.7311)\approx 2.718\). The baseline odds are \(0.10/0.90\approx0.1111\), so
Thus this candidate has about 24.5 times the baseline prior odds of inclusion. The raw score 0.80 is not the probability; the fitted calibration function produces the probability. The hypothetical coefficients are usable only if the validation procedure and its assumptions support them for this target population.
Estimating and Checking the Mapping
For the logistic model, the validation labels are treated as Bernoulli outcomes. Under the model, the likelihood for the calibration parameters is
Taking logarithms turns the product into a sum, yielding the log-likelihood \(\ell(a,b)=\sum_i[y_i\log\{\pi(s_i)\}+(1-y_i)\log\{1-\pi(s_i)\}]\). Maximizing it estimates the intercept and slope. This is a calibration fit, not evidence that the score causes an interaction. For the fit to estimate target probabilities, the validation labels must be sufficiently accurate, the validation candidates must represent the intended application or be appropriately adjusted, and the fitted functional form must be adequate. If candidates were selected for validation based on score or prior knowledge, unadjusted proportions may not represent the target population.
Calibration should be assessed on observations not used to fit the mapping, or with a suitable resampling procedure when data are limited. A reliability table or plot groups predictions and compares each group’s mean predicted probability with its observed positive fraction. The Brier score is the mean squared difference between predicted probabilities and binary labels, \(\frac{1}{m}\sum_i(\hat{\pi}_i-y_i)^2\); smaller values indicate better overall probabilistic predictions for the evaluated data, though the score also reflects how separable the cases are. A high area under a ranking curve can coexist with poor calibration: it assesses ordering, not whether a reported probability matches an observed frequency.
Worked Example: A Score That Ranks but Is Not a Probability
In a hypothetical validation sample, candidates in three score bands have the following reference labels. Assume the label definition and candidate sampling match the intended target.
| Score band | Validated candidates | Reference positives | Observed positive fraction |
|---|---|---|---|
| Low | 60 | 6 | 0.10 |
| Middle | 60 | 18 | 0.30 |
| High | 60 | 48 | 0.80 |
The observed fractions are \(6/60=0.10\), \(18/60=0.30\), and \(48/60=0.80\). A simple band-based calibration would assign these estimated probabilities to candidates in the respective bands. The bands preserve the score ordering, but their probabilities come from the validation frequencies, not from the labels “low,” “middle,” and “high” or from the score scale.
For a high-band candidate, the estimated prior odds are \(0.80/0.20=4\). For a middle-band candidate, they are \(0.30/0.70\approx0.4286\). Against a baseline inclusion probability of 0.10, whose odds are \(0.10/0.90\approx0.1111\), the corresponding relative weights are \(4/0.1111\approx36.0\) and \(0.4286/0.1111\approx3.857\). These are estimated from finite counts and therefore uncertain; the proportions should not be treated as exact truths. The example also illustrates why a numerical score cannot be read as a probability without checking it against relevant outcomes.
Calibration Methods and Their Trade-Offs
Logistic calibration is compact and can work well when a smooth, approximately linear log-odds relationship is plausible. It estimates few parameters, but can be systematically wrong if the true relationship bends, plateaus, or changes sharply. A bin-based mapping is easy to explain and allows nonlinearity, but depends on choices of bin boundaries and can be unstable when bins contain few candidates.
Isotonic calibration estimates a nondecreasing mapping without requiring a straight line on the log-odds scale. It is useful when scores have a defensible ordering but the probability relationship is not known to be logistic. It does not repair a score whose ordering is uninformative, and with limited validation data it can produce a stepwise, variable mapping. Neither method automatically corrects selection bias, mislabeled reference data, or a mismatch between validation and application populations.
The choice is therefore not “which transform makes the score look probabilistic?” It is which mapping is supported by representative validation evidence and adequate held-out checks. If the only evidence supports ranking, retain the score as a ranking quantity rather than assigning it a probability interpretation. A prior weight can still be used as a deliberately chosen relative preference, but it should be described as such and examined for sensitivity; it is not a calibrated probability.
Worked Example: Checking Calibration on Held-Out Candidates
A hypothetical held-out set has ten candidates assigned predicted probability 0.80, of whom eight meet the reference definition, and ten assigned predicted probability 0.40, of whom four meet it. Calculate the observed fractions and the Brier score using these group totals.
The observed fractions are \(8/10=0.80\) and \(4/10=0.40\), matching the assigned probabilities in these two groups. For the first group, eight positive labels contribute \(8(0.80-1)^2=8(0.04)=0.32\); two negative labels contribute \(2(0.80-0)^2=2(0.64)=1.28\). Its total squared error is 1.60. For the second group, four positives contribute \(4(0.40-1)^2=4(0.36)=1.44\), and six negatives contribute \(6(0.40-0)^2=6(0.16)=0.96\), also totaling 2.40.
The group-level fractions agree with the predictions in this small held-out set, but the result does not prove perfect calibration in the target population: sampling uncertainty remains, and only two probability groups were examined. The Brier score summarizes squared probability errors, not calibration alone. A broader reliability assessment and a check that the held-out candidates represent the intended use are still needed.
Diagnostics, Failure Modes, and Limitations
A calibration mapping is most credible when the target event is explicit, the score is available without using the validation outcome, and the validation design represents the candidates to which the prior will be applied. Examine whether higher scores really correspond to higher positive fractions, whether the mapping extrapolates beyond the observed score range, and whether uncertainty is large in sparsely populated regions. If validation labels are clustered by protein, assay, or source, uncertainty calculations that treat every candidate as independent may be too optimistic.
Prevalence shifts deserve particular attention. If the fraction of true interactions differs between validation and application populations, the same score may not imply the same probability. In some settings an intercept adjustment can address a known prevalence change while retaining a justified slope, but this requires assumptions about how the populations differ; it is not a universal correction. Also check for incomplete reference labels: absence of a recorded interaction may mean “not yet observed,” rather than a reliable negative. Treating such cases as true negatives can distort calibration.
- Overinterpreting the score scale. A score between 0 and 1 is not automatically a probability. Compare predicted values with observed frequencies for the specified target.
- Confusing ranking with calibration. Correct ordering does not ensure that probabilities are accurate. Assess both ordering and probability agreement for their separate purposes.
- Ignoring the validation sample design. A score-selected or unusually well-studied validation set may not estimate probabilities for the intended candidate population.
- Extrapolating beyond observed scores. A fitted function can return a number outside the validation range, but that does not make the estimate well supported.
The calibrated probability is a prior input, not a posterior conclusion about a candidate after analyzing current omics data. Its interpretation depends on the reference labels and population, and finite validation data make the mapping uncertain. A single fitted probability also suppresses that calibration uncertainty unless it is explicitly carried into the model. These limitations are reasons to report the mapping and validate its sensitivity, not reasons to disguise a ranking score as a probability.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A confidence score ranges from 0 to 1. What additional evidence is needed before interpreting a score of 0.7 as a prior inclusion probability?
- A fitted model is \(\operatorname{logit}\{\pi(s)\}=-2+3s\), with raw score \(s=0.5\). Calculate \(\pi(s)\).
- If a calibrated inclusion probability is 0.25 and the baseline probability is 0.10, calculate the relative prior odds weight.
- Why can a score rank candidates well but still be poorly calibrated?
Solutions. (1) A defined target event and relevant validation data showing that scores near 0.7 correspond to an observed positive fraction near 0.7; the range alone is insufficient. (2) The log-odds are \(-2+3(0.5)=-0.5\), so \(\pi=1/[1+\exp(0.5)]\approx0.3775\). (3) Candidate odds are \(0.25/0.75=0.3333\), baseline odds are \(0.10/0.90=0.1111\), and the relative weight is \(0.3333/0.1111=3.000\). (4) Ranking checks whether positives tend to appear above negatives, whereas calibration checks whether predicted probabilities match observed frequencies; correct ordering alone does not ensure correct probability values.