Confidence Levels and Error Tolerances in Credibility

From a Precision Rule to a Probability Requirement

A credibility standard is often described using two quantities: a confidence level and an error tolerance. The confidence level states how often an estimate must meet the precision requirement under repeated sampling. The tolerance states how far the estimate may fall from its target while still counting as sufficiently accurate. Together, they define a probability event; neither number alone specifies the standard.

This tutorial focuses on translating those choices into the event that a model must satisfy. It does not determine how much data are needed to achieve the event. That requires a model for the estimator’s variability, which is a separate step.

Definition: Let $\theta>0$ be the target quantity and $\hat{\theta}$ an estimator of it. For a symmetric relative error tolerance $r$, the estimate is within tolerance when its absolute error is no greater than $r\theta$: $$ |\hat{\theta}-\theta|\le r\theta. $$ For a required confidence level $p$, the corresponding probability requirement is $$ P\bigl(|\hat{\theta}-\theta|\le r\theta\bigr)\ge p. $$ The probability is calculated under the sampling model for $\hat{\theta}$, with $\theta$ treated as the target.

A confidence level of $p=0.90$, for example, requires that at least 90% of the probability lie inside the tolerance bounds. Equivalently, the probability of falling outside those bounds must be no more than $1-p=0.10$. In a credibility application, this is the precise meaning of the chosen confidence-and-tolerance requirement.

Writing the Bounds Correctly

For a positive target and a symmetric tolerance $r$, divide the defining inequality by $\theta$, which is positive. The event can then be written as a range for the estimate:

|\hat{\theta}-\theta|\le r\theta \quad\Longleftrightarrow\quad -r\theta\le\hat{\theta}-\theta\le r\theta \quad\Longleftrightarrow\quad (1-r)\theta\le\hat{\theta}\le(1+r)\theta.

Thus, a 5% tolerance means that the estimate may be as low as 95% of the target or as high as 105% of the target. The tolerance is measured relative to $\theta$, not relative to the estimate. In these expressions, $r$ is a decimal: a 5% tolerance is $r=0.05$.

The interval is inclusive: an estimate exactly on either boundary is within tolerance. That detail can affect a discrete probability calculation. If $\hat{\theta}$ has probability mass at a boundary, that mass belongs in the success probability.

Identify the target and estimator.
Write $\theta$ for the positive target and $\hat{\theta}$ for the quantity whose sampling variation is being assessed.

Convert the tolerance to bounds.
For symmetric relative tolerance $r$, use $(1-r)\theta$ and $(1+r)\theta$. Keep the tolerance in decimal form.

State the required probability.
For confidence level $p$, require probability at least $p$ inside the bounds, or at most $1-p$ outside them.

Evaluate under the stated model.
Use the model for $\hat{\theta}$ to calculate the probability of the event. The confidence-and-tolerance choices do not themselves provide that model.

Why the Complement Gives the Failure Requirement

The success event is that the estimate lies inside the permitted interval. Its complement is that the estimate is below the lower bound or above the upper bound. These two failure events do not overlap, so their probabilities add.

\begin{aligned} P\bigl((1-r)\theta\le\hat{\theta}\le(1+r)\theta\bigr) &=1-P\bigl(\hat{\theta}<(1-r)\theta \text{ or }\hat{\theta}>(1+r)\theta\bigr)\\ &=1-P\bigl(\hat{\theta}<(1-r)\theta\bigr) -P\bigl(\hat{\theta}>(1+r)\theta\bigr). \end{aligned}

Therefore, requiring confidence at least $p$ is equivalent to requiring the sum of the two tail probabilities to be no greater than $1-p$. This equivalence explains why a confidence level is not itself a tail probability. For example, 90% confidence permits at most 10% total failure probability, not 10% in each tail.

Worked Example: Testing a Symmetric Count Tolerance

A hypothetical insurer uses an estimate $\hat{\theta}$ of a target annual claim count $\theta=200$. Its exact sampling distribution is specified by the following probability masses: $\hat{\theta}=184,190,196,200,204,210,216$, with probabilities $0.04,0.08,0.18,0.40,0.18,0.08,0.04$, respectively. The requirement is 90% confidence within a 5% relative error tolerance. Does the distribution meet it?

Method: translate the tolerance, then sum the probability masses in the interval. The model here is the stated discrete distribution; no interest or mortality assumptions apply. First convert 5% to $r=0.05$. The lower bound is $200(1-0.05)=190$, and the upper bound is $200(1+0.05)=210$.

The qualifying estimates are 190, 196, 200, 204, and 210. Both endpoints qualify. Their total probability is

$$ 0.08+0.18+0.40+0.18+0.08=0.92. $$

As a check, the probabilities outside the interval are $0.04$ at 184 and $0.04$ at 216, for total failure probability $0.08$. Thus success probability is $1-0.08=0.92$, consistent with the direct sum. Since $0.92\ge0.90$, the requirement is met.

Common mistake avoided: omitting an endpoint because it is exactly 5% from the target. The event uses $\le$, so estimates at 190 and 210 are included.

Confidence Is a Total, Not an Automatic Tail Split

A two-sided requirement has a lower failure tail and an upper failure tail. The confidence level constrains their sum; it does not require them to be equal. Equal allocation is sometimes chosen as a design convention, in which case each tail receives at most $(1-p)/2$. But that allocation is an additional choice, not a consequence of the confidence level itself.

This distinction matters when the estimator’s distribution is skewed or discrete. The probability below the lower bound may differ substantially from the probability above the upper bound. To check the overall requirement, calculate both tails and add them, or calculate the probability inside the interval directly. Do not infer a symmetric distribution merely because the tolerance bounds are symmetric around the target.

Worked Example: A Stated Model That Misses the Confidence Level

A hypothetical estimate $\hat{\theta}$ is used for a target loss cost of $\theta=\$800$ per exposure. Its exact discrete distribution assigns probabilities $0.02,0.08,0.22,0.35,0.28,0.05$ to estimates of $\$720,\$740,\$780,\$820,\$860,\$880$, respectively. The selected tolerance is 7.5%, and the required confidence is 95%. Test the requirement.

Method: calculate the inclusive interval and compare its probability with the required confidence. The model is the stated probability distribution. There is no discounting or mortality in this calculation. The tolerance is $r=0.075$, so the lower bound is $800(0.925)=\$740$, and the upper bound is $800(1.075)=\$860$.

The estimates inside the interval are $\$740,\$780,\$820,\$860$. The success probability is

$$ 0.08+0.22+0.35+0.28=0.93. $$

The total failure probability is $0.02+0.05=0.07$, so the complementary check gives $1-0.07=0.93$. The requirement is 0.95, and $0.93<0.95$. Therefore, this model does not meet the selected confidence-and-tolerance standard.

Common mistake avoided: calling the requirement satisfied merely because the confidence is “close” to 95%, or confusing the 7.5% tolerance with an allowable failure probability. The probability must be compared directly with the specified 95% requirement.

Asymmetric Tolerances and One-Sided Requirements

Not every precision rule needs to treat overestimation and underestimation alike. If the permitted relative error below the target is $r_L$ and the permitted relative error above the target is $r_U$, the event is

(1-r_L)\theta\le\hat{\theta}\le(1+r_U)\theta.

The confidence requirement remains a probability statement about this specified interval. Its lower and upper limits need not be equally far from the target, and the probability in the two failure tails need not be equal. A one-sided requirement is also possible, such as requiring only that $\hat{\theta}$ not exceed a specified upper limit with probability at least $p$. In that case, state the one-sided event explicitly; do not silently replace it with a two-sided criterion.

Worked Example: Checking Different Lower and Upper Tolerances

A hypothetical pricing estimate $\hat{\theta}$ targets $\theta=1.25$. Its exact discrete distribution assigns probabilities $0.06,0.10,0.25,0.40,0.12,0.07$ to values $1.18,1.20,1.24,1.30,1.325,1.34$, respectively. The permitted error is 4% below the target and 6% above it. The requirement is 90% confidence. Does it pass?

Method: use the separate lower and upper tolerances. The distribution is given directly, with no interest or mortality assumptions needed. The lower bound is $1.25(1-0.04)=1.20$, and the upper bound is $1.25(1+0.06)=1.325$.

The qualifying values are $1.20,1.24,1.30,1.325$. Therefore, the success probability is

$$ 0.10+0.25+0.40+0.12=0.87. $$

The excluded values have probabilities $0.06+0.07=0.13$, so the complement check gives $1-0.13=0.87$. Since $0.87<0.90$, the model does not meet the requirement.

Common mistake avoided: applying a symmetric 4% tolerance just because the lower tolerance is 4%. The upper tolerance is explicitly 6%, so the correct upper limit is 1.325, not $1.25(1.04)=1.30$.

Common Exam Traps and Practical Interpretation

Using the wrong denominator. Relative tolerance is defined here against the target $\theta$: the error bound is $r\theta$. Do not replace it with $r\hat{\theta}$.
Converting confidence into the wrong failure probability. A confidence level $p$ permits total failure probability $1-p$. For a two-sided rule, do not assign $1-p$ to each tail unless the stated criterion explicitly does so.
Excluding boundary values. If the condition is written with $\le$, both bounds are included, including any discrete probability mass on them.
Assuming a model from the tolerance alone. Confidence and tolerance define the event but do not specify the distribution of $\hat{\theta}$. A probability calculation requires a sampling model or a stated distribution.
Confusing precision with adequacy. Passing a confidence-and-tolerance criterion says that the estimate meets the specified precision rule under the model. It does not establish that the target is an adequate rate or that the model is appropriate.

In practice, state the target, estimator, tolerance bounds, probability model, and required confidence together. This makes the criterion auditable and helps separate the chosen precision standard from the later task of determining what volume of data could satisfy it. The distinction is aligned with the CAS Exam 5 (Basic Ratemaking and Estimated Claim Liabilities) syllabus.

Key takeaway: For positive target $\theta$, symmetric relative tolerance $r$, and confidence requirement $p$, the standard is $P((1-r)\theta\le\hat{\theta}\le(1+r)\theta)\ge p$. Calculate that probability under the stated model, include the boundaries, and compare it directly with $p$.

Check Your Understanding

For each hypothetical model, translate the tolerance into bounds, evaluate the probability of meeting the criterion, and compare it with the required confidence. No interest or mortality assumptions apply to these probability calculations.

A target is $\theta=500$, with a 3% symmetric tolerance and 95% required confidence. The estimate takes values $480,485,500,515,520$ with probabilities $0.04,0.06,0.72,0.10,0.08$, respectively. Does it meet the standard? (Answer: No; the bounds are 485 and 515, and the success probability is $0.06+0.72+0.10=0.88<0.95$.)
A target is $\theta=1{,}200$, with a 10% symmetric tolerance and 90% required confidence. The estimate takes values $1{,}080,1{,}100,1{,}150,1{,}250,1{,}320,1{,}340$ with probabilities $0.08,0.10,0.25,0.30,0.12,0.15$, respectively. Does it meet the standard? (Answer: No; the bounds are 1,080 and 1,320, and the success probability is $0.08+0.10+0.25+0.30+0.12=0.85<0.90$.)
A target is $\theta=400$, with a 2% lower tolerance, a 5% upper tolerance, and 90% required confidence. The estimate takes values $390,392,400,418,420,421$ with probabilities $0.05,0.10,0.50,0.25,0.05,0.05$, respectively. Does it meet the standard? (Answer: Yes; the bounds are 392 and 420, and the success probability is $0.10+0.50+0.25+0.05=0.90$.)
A two-sided standard requires 99% confidence. What is the maximum total failure probability? If equal tail allocation is selected, what maximum is assigned to each tail? (Answer: The total failure probability is $1-0.99=0.01$; equal allocation assigns $0.005$ to each tail.)
A target is $\theta=250$, with an 8% symmetric tolerance and 95% required confidence. The estimate takes values $229,230,240,260,270,271$ with probabilities $0.02,0.07,0.20,0.40,0.22,0.09$, respectively. Does it pass? (Answer: No; the bounds are 230 and 270, and the success probability is $0.07+0.20+0.40+0.22=0.89<0.95$.)

What You'll Learn