Leave-One-Source-Out Calibration Checks

Why Omit One Data Source at a Time?

A model calibrated to several data sources can fit the combined evidence while depending heavily on only one source. A leave-one-source-out check makes that dependence visible: remove one source, refit the same model to the remainder, and compare the resulting parameters and predictions with the full-data fit. This is a sensitivity analysis of the calibration, not a new estimate based on an independent study.

In QSP, a “source” might be an assay, an experimental protocol, a study, or a collection of measurements whose shared design makes it reasonable to omit them as a unit. The choice matters. Leaving out one measurement at a time may fail to reveal dependence on an entire study if its measurements are correlated or share a protocol-specific feature.

The loss contributions may be negative log-likelihood terms or another specified calibration objective. Their scale and weighting determine how much each source contributes. A source with many observations, small assumed measurement error, or a deliberately larger weight can dominate the objective even if its raw residuals are not especially large. Therefore, report the objective and weighting convention used; “leave one out” does not itself prescribe either.

Interpretation also depends on the model assumptions. For a likelihood interpretation, the source contributions should correspond to an appropriate factorization of the data likelihood, conditional on the model and any shared parameters. If sources are dependent, simply summing independent-source likelihood contributions may be unjustified. A source-level omission can still be a useful sensitivity exercise, but its predictive interpretation must account for that dependence. Keep the model structure, data transformations, source weights, and fitting rules fixed across refits unless changing one is explicitly part of the question.

What to Compare

For each omitted source, examine both the refitted parameters and the predictions for that source. A parameter change identifies sensitivity of the fitted parameter values; the omitted-source prediction indicates whether the remaining calibration still anticipates the excluded evidence. Neither comparison alone is sufficient. A parameter may move substantially while predictions remain similar because of parameter compensation, or a small parameter shift may produce a large prediction change in a sensitive part of the model.

Here \(\hat{\theta}\) is the full-data estimate and \(\hat{\theta}_{-s}\) is the estimate omitting source \(s\). For a vector parameter, inspect components as well as any scientifically meaningful parameter combinations. Because components can have different units and scales, a raw change alone is not a universal measure of importance. Compare it with a meaningful scale, such as a prespecified parameter range or an uncertainty scale from the calibration, and state how that scale was chosen.

For held-out predictions, let \(\tilde{y}_{s,-s}\) denote the prediction for the omitted source using \(\hat{\theta}_{-s}\), and let \(y_s\) denote its observed data. Compare predictions and observations on the measurement scale, with units, and use a source-appropriate summary of predictive discrepancy. If the original source loss is appropriate for this purpose, its value evaluated at \(\hat{\theta}_{-s}\) can be reported as a held-out loss. Do not confuse it with the loss that was minimized in the refit: the omitted source did not contribute to that optimization.

Retaining a Prior or Penalty

If the original calibration includes a prior or penalty, omitting a data source should not silently remove that contribution. The full and leave-out objectives must be defined on the same basis, with only the selected source loss removed. In a Bayesian calibration, \(P(\theta)\) may represent the negative log prior; in a penalized fit, it may encode regularization. In either case, the leave-out estimate remains influenced by that contribution.

Why Some Sources Cause Larger Changes

A local approximation explains the role of a source’s contribution to the objective. Assume the objective is twice differentiable and the full estimate is an interior minimum, so its gradient is zero. Write \(g_s=\nabla L_s(\hat{\theta})\) and let \(H_{-s}\) be the Hessian matrix of the leave-out objective near the full estimate. At \(\hat{\theta}\), the gradient of the leave-out objective is \(-g_s\), because the full-objective gradient is zero and removing \(L_s\) subtracts its gradient. A first-order expansion of that gradient around \(\hat{\theta}\) gives the following approximation.

This approximation requires \(H_{-s}\) to be invertible and the refit to remain near the full estimate. It shows two reasons for a large shift: the omitted source has a large gradient at the full fit, or the remaining data constrain some parameter direction weakly, making the leave-out objective relatively flat. It is a diagnostic explanation, not a substitute for actually refitting. If the model is nonlinear, the optimum changes substantially, or the estimate is on a parameter boundary, the approximation may be poor.

Diagnostics, Alternatives, and Limitations

A useful report presents one row per source and includes the source omitted, parameter estimates or changes, held-out predictions, and an appropriate predictive discrepancy. Check that every refit converged to a meaningful solution and did not merely stop at a poor local optimum. If a fit fails, changes parameter bounds, or lands on a boundary, record that explicitly: it is evidence of instability, not a value to hide from the comparison.

Interpret the pattern rather than applying an automatic cutoff. Large parameter shifts can indicate that a source strongly constrains the estimate, but may also indicate weak identifiability in the remaining data. Large held-out errors identify sources that are poorly predicted by calibrations excluding them; they do not by themselves distinguish a discordant measurement from model misfit. Compare the predictions with source-specific residual patterns and known measurement characteristics, while keeping those explanations separate from what the omission calculation establishes.

Other calibration checks answer related but different questions. Residual inspection examines discrepancies in the fitted data; robust losses reduce the influence of large residuals according to a chosen loss function; and source-level heterogeneity models allow sources to differ through explicit model terms. Leave-one-source-out refitting instead asks how the calibration and its predictions change when one entire source contributes no data to the fit. These approaches can complement one another, but none automatically identifies a causal reason for a discrepancy.

The check has practical limitations. Results depend on how sources are grouped, on the chosen loss and weights, and on the model’s ability to extrapolate to the omitted source’s conditions. Shared participants, laboratories, or protocols can make sources dependent, so a nominally held-out source may not be independent of the remaining data. A large shift is a sensitivity signal rather than proof that the source is invalid; a small shift is not proof that its information is unimportant, particularly when other sources are redundant. If many sources are examined, summarize the full pattern rather than selecting only the most striking omission.

• Wrong reasoning: “The omitted source has a large prediction error, so it must be a bad measurement.”
• Correction: The error shows that the remaining-data fit predicts that source poorly under the specified model and objective. Measurement problems, model inadequacy, and source dependence are possible explanations that require additional evidence.
• Wrong reasoning: “The parameter changed by 0.5, so the source has a large influence.”
• Correction: A raw change has no universal meaning across parameters with different scales and units. Interpret it relative to a stated scientific or uncertainty scale and examine its implications for model predictions.