Assessing ABC Approximation Error

Why Assess Approximation Error?

Approximate Bayesian computation (ABC) makes calibration possible when a model can generate data but its likelihood is unavailable or impractical to evaluate. Its output is not automatically a close approximation to the posterior for the full data. It depends on the chosen summary statistics, the discrepancy measure, the tolerance, and the number and arrangement of simulations. A stable-looking posterior can still be stably approximating the wrong target.

Assessment therefore means varying these choices in a controlled way and asking whether the quantities that matter for the QSP decision are robust. The earlier material on rejection ABC, summary-statistic choice, tolerance selection, and sequential Monte Carlo ABC supplies the algorithms. Here the focus is how to evaluate what their output means. The earlier material on Synthetic Likelihood for Intractable QSP Models also offers a useful alternative approximation to compare when it is applicable.

Definition: Let $y_{\mathrm{obs}}$ be the observed data, $\theta$ a model parameter with prior density $\pi(\theta)$, and $S(y)$ a selected summary statistic. A common ABC posterior with tolerance $\varepsilon$ and distance $\rho$ is $$ \pi_{\varepsilon}(\theta\mid S(y_{\mathrm{obs}})) \propto \pi(\theta)\, P_{\theta}\!\left\{\rho\big(S(Y),S(y_{\mathrm{obs}})\big)\leq \varepsilon\right\}. $$ Here $Y$ is simulated from the model at $\theta$. This distribution conditions on a summary being sufficiently close, not necessarily on observing the full data.

Three distinct sources of change must be kept separate. Tolerance error arises because a nonzero $\varepsilon$ admits simulations that do not exactly match the observed summary. Summary error or information loss arises because $S(y)$ may discard features of the data relevant to $\theta$. Simulation error arises because probabilities and posterior summaries are estimated from a finite, possibly dependent set of simulations. Inadequate prior coverage, model misspecification, and numerical error can also affect calibration; increasing the simulation count does not repair those problems.

What Shrinking the Tolerance Can—and Cannot—Do

Suppose $S(Y)$ has a density $f_S(s\mid\theta)$, and use a distance region $B_\varepsilon(S_{\mathrm{obs}})$ around the observed summary. For small $\varepsilon$, continuity of this density gives the local approximation

P_\theta\{S(Y)\in B_\varepsilon(S_{\mathrm{obs}})\} = \int_{B_\varepsilon(S_{\mathrm{obs}})} f_S(s\mid\theta)\,ds \approx f_S(S_{\mathrm{obs}}\mid\theta)\,\operatorname{Vol}(B_\varepsilon).

For a fixed distance region whose volume does not depend on $\theta$, that common volume cancels when the posterior is normalized. Thus, under regularity conditions, as $\varepsilon$ tends to zero the ABC posterior approaches the posterior based on the summary likelihood $f_S(S_{\mathrm{obs}}\mid\theta)$. This is a justification for using a sequence of smaller tolerances to assess tolerance effects; it is not a guarantee that the result is the full-data posterior.

Important qualification: If $S$ is sufficient for $\theta$ under the specified model, conditioning on $S(y_{\mathrm{obs}})$ retains the likelihood information about $\theta$. Otherwise, even the zero-tolerance limit can differ from the posterior based on the full data. A smaller tolerance reduces one approximation, not every source of approximation.

A practical comparison holds the prior, model, summaries, distance scaling, and algorithm fixed while changing $\varepsilon$. Compare posterior means or medians, uncertainty intervals, parameter correlations, and probabilities tied to scientific decisions—not merely acceptance rates. Use the same reported units and scales in each run. Where possible, use repeated independent runs or enough simulations to determine whether the observed differences exceed simulation uncertainty.

Worked Example: Checking Sensitivity to Tolerance

Hypothetical model. A scalar parameter $\theta$, measured in clearance units, has a uniform prior from 0 to 10. The observed summary is 4.0, and simulated summaries satisfy $S(Y)\sim N(\theta,1)$. Rejection ABC uses the absolute difference $|S(Y)-4.0|$ and accepts when that difference is at most $\varepsilon$. Three runs each make 20,000 independent prior simulations.

Suppose the runs yield 3,180 accepted simulations at $\varepsilon=1.0$, 804 at $\varepsilon=0.25$, and 160 at $\varepsilon=0.05$. The acceptance rates are $3180/20000=0.159$, $804/20000=0.0402$, and $160/20000=0.00800$, respectively. The decreasing rates are expected: the acceptance event becomes more restrictive.

Suppose the corresponding accepted-sample means and standard deviations are 4.01 and 1.16, 4.00 and 1.01, and 3.99 and 1.00 clearance units. The posterior means are nearly unchanged, while the spread narrows as the tolerance shrinks. This pattern is consistent with tolerance affecting uncertainty more visibly than location in this symmetric example. It does not establish that the remaining posterior is exact; with $\varepsilon=0.05$, only 160 draws were accepted, so simulation error still deserves attention.

The simulation model and prior make the interpretation checkable: with a flat prior away from its boundaries, the limiting summary-based posterior is approximately normal around 4.0 with standard deviation 1.0. The smallest-tolerance result is close to that limit. In a real QSP calibration, the corresponding benchmark is usually unknown, so the evidence comes from stability across tolerances, simulation precision, and model-specific predictive checks.

Summary Statistics Define the Information Being Calibrated

Changing the summary statistic changes the target of ABC, not just the accuracy with which one fixed target is estimated. A summary designed to match a mean may fail to distinguish parameter combinations that produce different time-course shapes. Conversely, adding summaries can help retain relevant information, but it can also make close matches rarer and distances more sensitive to scaling. Summaries should therefore represent scientifically relevant features of the observations, with their scaling and distance specified explicitly.

To assess summary dependence, compare scientifically defensible sets using the same prior and model, and report the chosen summaries, scaling, distance, tolerance-selection rule, and simulation effort for each. A raw numerical change across different summary sets should not be described as a tolerance effect. If posterior conclusions differ, examine which data features each set uses and whether the difference affects predictions or decisions of interest.

Worked Example: Comparing Two Summary Sets

Hypothetical QSP calibration. A model has a receptor-turnover parameter $\theta$, measured in hours, and the observed data are a concentration time course. Analysts compare two ABC summary sets at the same prior, scaled-distance rule, tolerance-selection approach, and simulation budget. Set A contains the mean concentration and the final concentration. Set B contains those two summaries plus peak concentration and time to peak.

Suppose each run produces 1,000 effective posterior draws. For a prespecified scientific threshold of 8 hours, the estimated probability $P(\theta>8\text{ hours}\mid\text{ABC summaries})$ is 0.31 with Set A and 0.68 with Set B. The absolute difference is $|0.68-0.31|=0.37$, or 37 percentage points. This is a substantial change in the threshold conclusion, not evidence that one posterior is automatically correct.

The comparison suggests that peak-related features contain information about $\theta$ not captured by the mean and final concentration alone, or that the new summaries alter the effective matching problem. The next diagnostic is to check whether the observed peak and time to peak are reproduced by simulations from each calibrated posterior, and whether the change persists across reasonable tolerances and independent runs. If the scientific conclusion depends on the threshold, Set A alone is not a robust basis for that conclusion.

The added summaries are not guaranteed to improve inference merely because there are more of them. If they are noisy, redundant, poorly scaled, or weakly connected to $\theta$, they can reduce acceptance without adding useful information. The comparison is informative precisely because it tests a scientifically motivated alternative and records what changed.

Simulation Error: Precision Is Not Accuracy

For independent rejection ABC, accepted parameter draws are draws from the ABC posterior conditional on acceptance. If $M$ draws are accepted and $g(\theta)$ is a scalar quantity of interest, the Monte Carlo estimate is $\hat{g}=M^{-1}\sum_{i=1}^{M}g(\theta_i)$. Its estimated Monte Carlo standard error is $s_g/\sqrt{M}$, where $s_g$ is the sample standard deviation of the accepted $g(\theta_i)$ values. For a posterior probability, use the accepted proportion and its binomial standard error. These quantify simulation variability, not discrepancy between the ABC posterior and the desired full-data posterior.

In SMC ABC, particles are dependent and may have unequal weights. The raw particle count is not generally the number of independent posterior draws. Use an appropriate weighted effective sample size, repeated independent runs, or another valid Monte Carlo error assessment for the algorithm. A large particle count with severe weight concentration can provide little precision. Compare Monte Carlo uncertainty with the changes caused by varying $\varepsilon$ or $S$: if the changes are smaller than simulation uncertainty, the comparison is inconclusive.

Worked Example: Is a Posterior Probability Estimated Precisely?

Hypothetical rejection ABC runs. In one run, 5,000 prior simulations yield 250 accepted draws. Of those, 146 have $\theta>8$ hours. The estimated posterior probability is

\hat{p}=\frac{146}{250}=0.584,\qquad SE(\hat{p})\approx \sqrt{\frac{0.584(1-0.584)}{250}}=0.0312.

Thus, the estimated probability is 58.4%, with Monte Carlo standard error about 3.12 percentage points under the independent-draw approximation. A rough 95% Monte Carlo range is $0.584\pm1.96(0.0312)$, or 0.523 to 0.645. This range describes simulation precision for the estimated ABC probability; it does not include tolerance or summary-statistic error.

A second independent run with 20,000 proposals yields 1,000 accepted draws, including 584 above 8 hours. Its estimate is $584/1000=0.584$, and its estimated standard error is $\sqrt{0.584(0.416)/1000}=0.0156$. The acceptance rate is 5% in both runs, while the second run has four times as many accepted draws and half the standard error, approximately. Agreement between the estimates is reassuring about simulation variability, but both runs use the same ABC target and therefore can share the same approximation error.

A Controlled Assessment Workflow

Define the target of assessment.
Choose posterior quantities that matter, such as a parameter interval, a threshold probability, or a prediction from the calibrated QSP model. State units and decision thresholds before comparing runs.

Assess simulation precision.
At a fixed summary and tolerance, increase simulation effort or run independent replicates. Record accepted draws or effective sample size, Monte Carlo errors, and any evidence of particle weight concentration.

Assess tolerance sensitivity.
Hold the summaries, scaling, and algorithm fixed while varying tolerance. Compare posterior quantities and predictive behavior, and check whether apparent changes exceed Monte Carlo uncertainty.

Assess summary sensitivity.
Compare defensible summary sets while documenting every change in summaries, distance, and scaling. Investigate which data features explain differences in posterior conclusions.

Report limitations and robustness.
Distinguish findings stable across tested settings from those that depend on the ABC choices. Describe the checks performed; do not imply that a finite set of comparisons proves exactness.

Diagnostics, Alternatives, and Limitations

Useful diagnostics include acceptance rate, effective sample size, Monte Carlo errors, stability across independent runs, and the behavior of scientific posterior quantities across tolerances. Also check whether simulations from the fitted parameter region can reproduce the observed summaries and important aspects of the data not used for matching. If an observed summary lies in a region rarely reached by simulations under the prior, acceptance may be extremely low; this can reflect poor prior coverage, an implausible model, an unsuitable summary, or insufficient simulation. It is not, by itself, proof of which explanation is responsible.

Approach	What it can diagnose	What it does not establish
More simulations at fixed ABC settings	Whether Monte Carlo variability is affecting estimated ABC quantities	Whether the selected summaries or tolerance approximate the full-data posterior well
Smaller tolerances at fixed summaries	Sensitivity to the nonzero-tolerance approximation	Whether the summaries retain all information relevant to the parameters
Alternative summary sets	Sensitivity to the information encoded by the summaries	That the richer or more stable set is necessarily sufficient or correct
Synthetic likelihood	An alternative likelihood approximation based on a model for simulated summaries	Freedom from approximation error; its assumptions and simulation precision also require assessment

A common mistake is to increase the number of simulations and conclude that ABC approximation error has been solved. More simulations can reduce Monte Carlo error, but the ABC target still depends on the summaries and tolerance. The correction is to report simulation uncertainty separately from sensitivity to those choices. Another mistake is to treat a small tolerance as proof of accuracy: a summary can omit important information even when it is matched almost exactly.

No finite sensitivity analysis proves that an ABC posterior equals the full-data posterior. Computation can also limit the smallest usable tolerance: acceptance falls, SMC particles can degenerate, and summary distances may be difficult to compare in high dimensions. Conclusions should therefore be calibrated to the evidence: report which results are robust across tested settings and which remain conditional on a particular approximation.

Key takeaway: Evaluate ABC approximation by separating the questions: Is simulation precision adequate? Are conclusions sensitive to tolerance? Do plausible summaries lead to different conclusions? Stable answers across these checks support robustness for the tested choices, but do not certify exact inference.

Check Your Understanding

Answer each question, then compare with the solutions.

At fixed summaries, an ABC posterior probability changes from 0.52 to 0.55 as the tolerance shrinks. What additional information is needed before deciding whether this change is meaningful?
Why does increasing the number of prior simulations not correct information lost through an insufficient summary statistic?
In a rejection ABC run, 400 of 8,000 simulations are accepted. What is the acceptance rate, and what does it tell you?
Two summary sets produce different posterior probabilities for a mechanistic threshold. Give one appropriate interpretation and one claim that would be unjustified.

Solutions. (1) Estimate Monte Carlo uncertainty for both probabilities, using accepted independent draws or an appropriate method for dependent or weighted output. A 0.03 difference is meaningful only relative to that uncertainty and the scientific decision. Also verify that the runs differ only in tolerance. (2) More simulations improve precision for the chosen ABC target; they cannot restore data features discarded before matching. (3) The rate is $400/8000=0.05$, or 5%. It describes how often proposals meet the specified acceptance rule and helps assess computational efficiency; it does not measure posterior accuracy. (4) The difference indicates sensitivity to the information represented by the summaries and warrants investigation of the relevant data features and robustness. It would be unjustified to claim, without further evidence, that the richer summary is correct or that either result equals the full-data posterior.