Deviation Events and Confidence Levels for Risk Estimates

Why Deviation Events Matter

A risk estimate is useful only when we can describe how far it might be from the quantity it estimates. The central question is not merely whether an estimate is close in one observed sample, but how often the estimation error would exceed a chosen tolerance across repeated samples. A deviation event expresses that question precisely.

We focus on a sample average. Let $Z_1,\ldots,Z_n$ be independent observations from the same distribution, with finite mean $R=E[Z_1]$. In a risk-estimation setting, each $Z_i$ can represent a loss on a randomly sampled case, and $R$ is the expected loss. The sample-average estimate is $\widehat R=n^{-1}\sum_{i=1}^n Z_i$. The probability statements below refer to the randomness in repeatedly drawing the sample. They do not, by themselves, say that a particular realized estimate is close to $R$.

Definition: For a tolerance $\varepsilon>0$, the two-sided deviation event is $$ \{|\widehat R-R|>\varepsilon\}. $$ Its probability $P(|\widehat R-R|>\varepsilon)$ is the tail probability at tolerance $\varepsilon$. A bound $P(|\widehat R-R|>\varepsilon)\leq\delta$ says that the probability of an error larger than $\varepsilon$ is at most $\delta$; equivalently, the estimate is within $\varepsilon$ with probability at least $1-\delta$.

The three quantities have different roles. The tolerance $\varepsilon$ measures the size of an error in the units of risk. The number $\delta$ is an upper bound on the probability of exceeding that tolerance. The corresponding confidence level is $1-\delta$, a probability of being within tolerance. For example, $\delta=0.05$ corresponds to a confidence level of at least $0.95$, or 95%.

From a Tail Bound to a Confidence Statement

The translation is an event-complement identity, not an approximation. The event that the error is at most $\varepsilon$ is exactly the complement of the event that it is greater than $\varepsilon$:

\{|\widehat R-R|\leq\varepsilon\} = \{|\widehat R-R|>\varepsilon\}^{\,c}.

Taking probabilities gives the exact relationship below. The result requires only that $\widehat R$ and $R$ be defined so that the events are measurable; it does not require independence, identical distributions, or a particular distribution for the observations.

Result: For any $\varepsilon\geq 0$ and $0\leq\delta\leq 1$, $$ P(|\widehat R-R|>\varepsilon)\leq\delta \quad\Longleftrightarrow\quad P(|\widehat R-R|\leq\varepsilon)\geq 1-\delta. $$

A two-sided deviation event can also be separated into its two tails. The strict inequalities matter: an error exactly equal to $\varepsilon$ is not in the event $\{|\widehat R-R|>\varepsilon\}$, but is included in the event $\{|\widehat R-R|\leq\varepsilon\}$.

Result: For every $\varepsilon\geq0$, $$ P(|\widehat R-R|>\varepsilon) = P(\widehat R-R>\varepsilon)+P(\widehat R-R<-\varepsilon). $$ In particular, if the two one-sided probabilities are each at most $\delta/2$, then the two-sided tail probability is at most $\delta$.

Proof. The event $\{|\widehat R-R|>\varepsilon\}$ is the union of $\{\widehat R-R>\varepsilon\}$ and $\{\widehat R-R<-\varepsilon\}$. These events are disjoint: a real number cannot be both greater than $\varepsilon$ and less than $-\varepsilon$. Additivity of probability for disjoint events gives the equality. If each probability on the right is at most $\delta/2$, their sum is at most $\delta$. $\square$

The converse of the final sentence does not generally hold: a two-sided bound of $\delta$ does not require the two tails to be equal or each to be at most $\delta/2$. One side could account for most of the tail probability. Symmetry of the distribution of $\widehat R-R$, when it holds, can make the tails equal, but that is an additional property, not part of the definition of a two-sided guarantee.

Worked Examples

Worked Example: An Exact Bernoulli Tail Probability

Suppose each observed loss $Z_i$ is Bernoulli with success probability $R=0.25$, and $n=4$. Find the probability that the sample-average estimate differs from $R$ by more than $\varepsilon=0.25$, then state the resulting confidence level.

Let $K=\sum_{i=1}^4 Z_i$. Then $\widehat R=K/4$, and $K$ has a binomial distribution with four trials and success probability $0.25$. The deviation event is $|K/4-0.25|>0.25$. If $K=0$, the absolute error is exactly $0.25$, so this value is excluded by the strict inequality. If $K=1$ or $K=2$, the error is respectively $0$ or $0.25$, also not greater than the tolerance. If $K=3$ or $K=4$, the error is respectively $0.5$ or $0.75$, so those values are included.

Consequently, $P(|\widehat R-R|>0.25)=P(K=3)+P(K=4)$. The two probabilities are $P(K=3)=\binom{4}{3}(0.25)^3(0.75)=12/256$ and $P(K=4)=\binom{4}{4}(0.25)^4=1/256$. Therefore the tail probability is $13/256$, approximately $0.0508$. The probability of an error no greater than $0.25$ is $1-13/256=243/256$, approximately $0.9492$. Thus the estimate is within $0.25$ with about 94.92% probability.

Worked Example: A Confidence Level Is Not an Error Tolerance

Let $Z_1,Z_2$ be independent Bernoulli observations with mean $R=0.5$. Take $\varepsilon=0.25$. Determine the tail probability and confidence level exactly.

The sample average takes the values $0$, $0.5$, and $1$. The probabilities are $P(\widehat R=0)=1/4$, $P(\widehat R=0.5)=1/2$, and $P(\widehat R=1)=1/4$. At $\widehat R=0.5$, the error is zero. At either endpoint, the absolute error is $0.5$, which exceeds $0.25$. Hence $P(|\widehat R-R|>0.25)=1/4+1/4=1/2$.

The probability of being within tolerance is $1-1/2=1/2$, so this is a 50% confidence guarantee at tolerance $0.25$. The number $0.25$ is not the confidence level, and $0.5$ is not the error tolerance: one is an error measured on the risk scale, while the other is a probability.

Worked Example: A Normal Sample Average

Suppose $Z_1,\ldots,Z_4$ are independent with distribution $N(R,1)$, and consider $\varepsilon=1$. Calculate the exact two-sided tail probability and the corresponding confidence level.

The average of independent normal observations is normal. Its mean is $R$, and its variance is $1/4$, so $\widehat R-R\sim N(0,1/4)$. Dividing by its standard deviation $1/2$ gives a standard normal variable $W=2(\widehat R-R)$. Therefore $P(|\widehat R-R|>1)=P(|W|>2)$.

Writing $\Phi$ for the cumulative distribution function of $N(0,1)$, symmetry gives $P(|W|>2)=P(W>2)+P(W<-2)=2\Phi(-2)$, approximately $0.0455$. The probability of an error at most $1$ is therefore $1-2\Phi(-2)$, approximately $0.9545$. The estimate is within one unit of $R$ with about 95.45% probability under this model.

How to Read a Risk-Estimation Guarantee

A statement such as $P(|\widehat R-R|>\varepsilon)\leq\delta$ is meaningful only when its target and probability are clear. Here $R$ is the expected value being estimated, $\widehat R$ is the random sample average, and the probability is taken over the sampling process. The statement guarantees a probability of accuracy across repeated samples; it does not assert that every sample gives an accurate estimate.

It is also important to keep the direction of the inequality straight. A small $\delta$ means a small allowed failure probability and thus a high confidence level. A small $\varepsilon$ asks for tighter accuracy, which is a stronger demand. Neither number can be inferred from the other without information about the distribution or a result that controls the tail probability. Defining the event does not itself prove that its probability is small.

Identify the target and estimate.
Write down $R$ and $\widehat R$, and identify what randomness the probability refers to.

Read the tolerance.
Interpret $\varepsilon$ in the units of risk: it is the maximum error allowed in the accuracy event.

Translate the tail probability.
If the tail probability is at most $\delta$, state that the estimate is within $\varepsilon$ with probability at least $1-\delta$.

Common Pitfall: Confusing Failure Probability with Confidence

A frequent mistake is to read $P(|\widehat R-R|>\varepsilon)\leq0.05$ as a 5% confidence statement. That probability concerns failure—an error larger than the tolerance—so the corresponding accuracy guarantee is at least 95%, not 5%. Another mistake is to treat the guarantee as a claim that the realized estimate has probability $1-\delta$ of being close after the data have already been observed. The probability statement describes repeated samples under the assumed sampling model; after observing a particular estimate, its numerical error is fixed, though $R$ may not be known.

Finally, “within $\varepsilon$” means $|\widehat R-R|\leq\varepsilon$, including equality. The failure event uses $>\varepsilon$, not $\geq\varepsilon$. This distinction can change an exact probability for discrete sample averages, as the Bernoulli calculation above demonstrates.

Key takeaway: A deviation guarantee has three separate components: an estimate, an error tolerance $\varepsilon$, and a failure-probability bound $\delta$. The equivalent confidence statement is that the estimate lies within $\varepsilon$ of its target with probability at least $1-\delta$.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

If $P(|\widehat R-R|>0.1)\leq0.02$, what is the corresponding minimum confidence level, and what error tolerance does it use?
Why does the event $\{|\widehat R-R|> \varepsilon\}$ exclude an error exactly equal to $\varepsilon$?
Can a two-sided tail bound of $\delta$ alone guarantee that each one-sided tail is at most $\delta/2$? Explain.
In the first worked example, why is $K=0$ excluded from the deviation event even though the estimate is not equal to $R$?
Does a deviation guarantee assert that every realized sample average is within the stated tolerance? Explain what the probability refers to.