Tolerance Selection in ABC

Why the ABC Tolerance Matters

In rejection approximate Bayesian computation (ABC), a simulated data set is retained when its summary statistics are sufficiently close to the observed summaries. The tolerance, $\varepsilon$, sets what “sufficiently close” means. A larger tolerance admits more simulations, but also admits simulations that may be less compatible with the observation. A smaller tolerance narrows the accepted set, but can leave too few parameter draws for stable posterior estimates.

Tolerance selection is therefore a balance, not simply a rule to choose the smallest possible number. A useful assessment compares posterior summaries over a decreasing sequence of tolerances, while tracking how many simulations are accepted and how uncertain the resulting estimates are. The summaries and discrepancy must be defined consistently across that sequence; in particular, rescaling summaries between runs would change the meaning of the same numerical tolerance.

Definition: Let $\theta$ denote the model parameters, with prior density $\pi(\theta)$. Let $S$ be the summary statistics from a simulated data set and $s_{\mathrm{obs}}$ the summaries from the observed data. For a discrepancy $d(S,s_{\mathrm{obs}})$, the hard-threshold ABC posterior at tolerance $\varepsilon$ is $$ \pi_{\varepsilon}(\theta\mid s_{\mathrm{obs}}) \propto \pi(\theta)\, P_{\theta}\!\left(d(S,s_{\mathrm{obs}})\leq\varepsilon\right). $$ It is the parameter distribution among prior draws whose simulated summaries fall within the tolerance. The probability is over simulated data conditional on $\theta$.

This expression assumes that parameters are drawn from the stated prior and that simulations are generated from the specified model. The discrepancy and summaries are fixed before comparing tolerances. With exact simulation and rejection sampling, the retained parameter draws represent the ABC posterior above, subject to Monte Carlo variability. ABC is approximate: even at very small tolerance, its limiting target is generally determined by the chosen summaries, not necessarily by the full-data posterior. If the summaries are sufficient for $\theta$, conditioning on them retains the information in the data relevant to $\theta$; otherwise, information can be lost.

How Tolerance Controls Acceptance and Approximation

For a fixed simulated parameter and data set, the event $d(S,s_{\mathrm{obs}})\leq\varepsilon$ can only become less likely when $\varepsilon$ is reduced. Thus the prior-predictive acceptance probability

A_{\varepsilon} = \int \pi(\theta)\, P_{\theta}\!\left(d(S,s_{\mathrm{obs}})\leq\varepsilon\right) \,d\theta

is nondecreasing in $\varepsilon$. If $N$ independent parameter-and-data simulations are run, the expected number accepted is $N A_{\varepsilon}$. The acceptance fraction estimates $A_{\varepsilon}$, but it is not a measure of posterior accuracy by itself: it also depends on the prior, the model, the summaries, their scaling, and the discrepancy.

The small-tolerance behavior explains why acceptance can fall sharply. Suppose $S$ is a continuous $q$-dimensional summary with conditional density $g_{\theta}(s)$, and the discrepancy is $d(S,s_{\mathrm{obs}})=\lVert W(S-s_{\mathrm{obs}})\rVert$, where $W$ is a fixed invertible scaling matrix. If $g_{\theta}$ is continuous near $s_{\mathrm{obs}}$, then for small $\varepsilon$,

P_{\theta}\!\left(d(S,s_{\mathrm{obs}})\leq\varepsilon\right) = \int_{\lVert W(s-s_{\mathrm{obs}})\rVert\leq\varepsilon} g_{\theta}(s)\,ds \approx V_q\varepsilon^q|\det W|^{-1}g_{\theta}(s_{\mathrm{obs}}),

where $V_q$ is the volume of the unit ball in $q$ dimensions. To see this, set $u=W(s-s_{\mathrm{obs}})$, so $ds=|\det W|^{-1}du$. The integral is then over $\lVert u\rVert\leq\varepsilon$. For small $\varepsilon$, continuity makes $g_{\theta}(s_{\mathrm{obs}}+W^{-1}u)$ close to $g_{\theta}(s_{\mathrm{obs}})$; the remaining integral is the ball’s volume, $V_q\varepsilon^q$, times the Jacobian factor. Integrating over the prior gives the same leading $\varepsilon^q$ dependence for $A_\varepsilon$.

This approximation requires a continuous summary distribution with positive, sufficiently regular density near the observed summaries. It is not a universal rate for every model or discrepancy. It does show why a tolerance reduction can be costly, especially when many summaries contribute to the distance. In the same small-tolerance limit, the factors $V_q\varepsilon^q|\det W|^{-1}$ cancel between the numerator and normalizing constant of the ABC posterior. Under these regularity conditions, the limit is proportional to $\pi(\theta)g_{\theta}(s_{\mathrm{obs}})$: the posterior based on the summaries.

Worked Example: Acceptance Rates Across Three Tolerances

In a hypothetical ABC calibration, 10,000 prior-predictive simulations are run once, with a fixed discrepancy. Of these, 1,200 have distance at most 2, 360 have distance at most 1, and 90 have distance at most 0.5. These counts are nested because every simulation accepted at a tighter tolerance is also accepted at a looser one.

The acceptance fractions are $1{,}200/10{,}000=0.12$, $360/10{,}000=0.036$, and $90/10{,}000=0.009$, or 12%, 3.6%, and 0.9%. Reducing the tolerance from 1 to 0.5 therefore leaves one quarter as many accepted draws in this run.

For the 0.9% acceptance estimate, a binomial standard error is approximately $ \sqrt{0.009(1-0.009)/10{,}000}=0.000944 $, or 0.0944 percentage points. This quantifies simulation uncertainty in the estimated acceptance fraction, not error in the posterior approximation. The practical consequence is clear: the tightest tolerance gives fewer parameter draws, so posterior summaries at that tolerance may be less precise unless more simulations are run.

Comparing Posterior Estimates Along a Tolerance Sequence

A sequence of tolerances makes the trade-off visible. For each tolerance, record the number accepted and compare relevant posterior quantities: parameter means or medians, intervals, correlations, and mechanistic outputs used in the scientific interpretation. The quantities should be chosen before inspecting the results where possible. A posterior can change in one parameter but remain stable for a clinically important model output, or the reverse.

A smaller tolerance does not guarantee that every estimate moves closer to the exact posterior, nor does a plateau prove exactness. Finite accepted samples fluctuate, and all tolerances share the limitations of the chosen summaries. Compare changes with their Monte Carlo uncertainty: a small shift relative to the uncertainty in both estimates is weak evidence of a meaningful tolerance effect, while a persistent shift larger than that uncertainty warrants investigation or more simulation.

Worked Example: Posterior Estimates for Clearance

Suppose a hypothetical QSP calibration estimates drug clearance in L/h from a fixed set of scaled summaries. The table reports posterior means and central 90% intervals from accepted draws at three tolerances. The accepted counts are from separate runs with the same prior, simulator, summaries, and discrepancy.

Tolerance $\varepsilon$	Accepted draws	Mean clearance (L/h)	Central 90% interval (L/h)
1.0	1,000	1.82	0.86 to 3.05
0.5	250	1.57	0.94 to 2.40
0.25	62	1.49	1.01 to 2.06

The mean decreases by $1.82-1.57=0.25$ L/h between tolerances 1.0 and 0.5, then by $1.57-1.49=0.08$ L/h between 0.5 and 0.25. The interval also narrows over this sequence. These results suggest that estimates are changing less at the tighter end, but they do not establish that 0.25 is preferable: only 62 draws were accepted there, so both its mean and interval may be unstable. A reasonable next step is to increase the number of simulations at 0.25 and check whether the estimate remains near 1.49 L/h. The reported interval describes posterior uncertainty under that ABC approximation; it does not include all uncertainty from using a finite tolerance or finite simulation.

The comparison is most informative when it separates three sources of change: the target distribution changes with $\varepsilon$; finite accepted samples produce Monte Carlo error; and choices made before the run, such as summary statistics and discrepancy scaling, determine what information ABC uses. Repeating simulations or increasing their number helps assess Monte Carlo variability. It does not repair uninformative summaries or an unsuitable discrepancy.

A Practical Selection Workflow

Fix the summaries and discrepancy.
Use the same observed summaries, scaling, distance, and simulator for every candidate tolerance. A numerical tolerance has no comparable meaning if these choices change.

Run a pilot and choose a sequence.
Inspect simulated distances to select tolerances that span a useful range of acceptance rates. The pilot helps plan computation; it does not determine a universally correct tolerance.

Track accepted counts and estimates.
For each tolerance, report the acceptance fraction, parameter summaries, intervals, and any mechanistic outputs central to the question. Include the number accepted so the reader can judge simulation precision.

Check stability and Monte Carlo uncertainty.
Compare successive estimates and, where changes could matter, run enough simulations to determine whether the pattern persists beyond sampling noise. Prefer a tolerance that supports stable conclusions at a feasible computational cost, rather than the smallest tolerance by default.

Worked Example: Choosing a Tolerance Under a Simulation Budget

In a hypothetical analysis, a pilot run indicates that tolerances 0.8, 0.4, and 0.2 give approximate acceptance rates of 8%, 2%, and 0.5%. The team can afford 100,000 simulations per candidate run. The expected accepted counts are therefore $100{,}000(0.08)=8{,}000$, $100{,}000(0.02)=2{,}000$, and $100{,}000(0.005)=500$.

For a model output measured in days, suppose the corresponding ABC posterior means are 14.2, 13.6, and 13.4 days. The shift from 0.8 to 0.4 is $14.2-13.6=0.6$ days; from 0.4 to 0.2 it is $13.6-13.4=0.2$ days. The tighter run may be useful if that 0.2-day change affects the scientific conclusion, but its 500 accepted draws make precision worth checking. If the conclusion is unchanged and additional simulation cannot be afforded, 0.4 may be a defensible computational choice, provided the result is reported as an ABC approximation at that tolerance. The acceptance rate alone cannot decide between them.

Diagnostics, Alternatives, and Limitations

Several warning signs indicate that tolerance comparisons need more care. A sharp shift in posterior estimates as $\varepsilon$ decreases can indicate substantial finite-tolerance effects, but it can also reflect Monte Carlo noise when few draws are accepted. Large changes between repeated runs at the same tolerance point to inadequate simulation for stable summaries. Very low acceptance means high computational cost per retained draw; high acceptance means many simulations are relatively far from the observed summaries. Neither rate alone diagnoses whether the summaries capture the relevant information.

The hard-threshold rule is not the only ABC approach. A kernel-weighted ABC method gives simulations closer to the observed summaries more weight rather than treating all simulations inside a cutoff equally. This changes the weighting scheme, but still requires a discrepancy, a bandwidth or tolerance, and enough simulation to estimate posterior quantities reliably. Exact likelihood-based Bayesian calibration avoids the ABC tolerance when a tractable likelihood is available and correctly specified; it is not an option merely because a smaller tolerance is computationally inconvenient. The choice among approaches depends on the model and inferential target.

Tolerance selection also has limits. No sequence of tolerances can recover information discarded by insufficient summaries, and an apparently stable sequence can share the same bias. A small acceptance fraction can make estimates noisy, while a large one can leave appreciable finite-tolerance approximation. Tolerance-specific posterior intervals do not automatically account for this approximation. State the summaries, discrepancy, tolerance, and accepted counts so that the scope of the result is clear.

Common pitfall: “The smallest tolerance must be the best because it makes the simulations most similar to the data.” A smaller tolerance does make the acceptance event stricter, but it can sharply reduce the number of draws and increase Monte Carlo error. It also cannot correct poor summaries or guarantee a closer estimate in a finite run. Compare posterior quantities across tolerances and assess their simulation precision before deciding.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why can the acceptance probability not increase when a hard-threshold tolerance is reduced, assuming the discrepancy and simulations are held fixed?
With 20,000 simulations, 400 are accepted at a particular tolerance. What is the observed acceptance fraction, and how many accepts would be expected from 50,000 simulations at the same rate?
In the continuous-summary setting, why does the factor involving the small tolerance cancel from the limiting ABC posterior?
A posterior mean changes little across three tolerances, but the tightest run has only 25 accepted draws. What should be checked before calling the estimate stable?
Does a stable tolerance sequence show that the summaries contain all the information in the data? Explain.

Solutions. (1) The accepted event at a smaller tolerance is a subset of the accepted event at a larger tolerance, so its probability cannot be greater. (2) The fraction is $400/20{,}000=0.02$, or 2%; at that rate, $50{,}000(0.02)=1{,}000$ accepts are expected. (3) For small tolerance, the leading factor $V_q\varepsilon^q|\det W|^{-1}$ multiplies the summary density for each $\theta$; the same factor appears in the normalizing integral over $\theta$, so it cancels. (4) Assess Monte Carlo variability, for example by running more simulations at that tolerance or repeating the run, and check whether the mean and other relevant summaries remain stable. (5) No. The sequence assesses sensitivity to the tolerance for the chosen summaries; all runs may omit information not represented by those summaries.

What You'll Learn