Bayes Rule Base-Rate and Normalization Pitfalls

Why Bayes-Rule Errors Are So Common

A conditional probability depends on what is given. The probability of seeing evidence \(E\) in class \(C\), \(P(E\mid C)\), answers a different question from the probability that the class is \(C\) after observing \(E\), \(P(C\mid E)\). Reversing these probabilities can produce a strikingly wrong conclusion, especially when the class is uncommon.

A reliable calculation keeps three quantities separate: the prior, which describes class probabilities before the evidence; the likelihood, which describes how probable the evidence is within each class; and the posterior, which describes class probabilities after observing the evidence. The prior probabilities weight the likelihoods, and the resulting weights must be normalized to sum to one.

Definition: Let \(C_1,\ldots,C_k\) be a finite partition of the sample space, with \(P(C_i)>0\) for each class considered. For an event \(E\) with \(P(E)>0\), the prior for class \(C_i\) is \(P(C_i)\), the likelihood of evidence \(E\) in that class is \(P(E\mid C_i)\), and the posterior is \(P(C_i\mid E)\).

The Formula and Its Normalization

The finite-class form of Bayes' rule follows by writing the joint probability of class \(C_i\) and evidence \(E\) in two ways. The denominator is the total probability of the evidence across all classes.

Theorem: If \(C_1,\ldots,C_k\) partition the sample space, \(P(C_i)>0\), and \(P(E)>0\), then

P(C_i\mid E) = \frac{P(E\mid C_i)P(C_i)} {\displaystyle\sum_{j=1}^k P(E\mid C_j)P(C_j)}.

Proof. By the definition of conditional probability,

P(C_i\mid E)=\frac{P(C_i\cap E)}{P(E)}.

Applying the same definition within class \(C_i\) gives \(P(C_i\cap E)=P(E\mid C_i)P(C_i)\). Since the \(C_j\) form a partition, the event \(E\) is the disjoint union of the events \(E\cap C_j\). Therefore,

P(E)=\sum_{j=1}^k P(E\cap C_j) =\sum_{j=1}^k P(E\mid C_j)P(C_j).

Substitution gives the stated formula. The denominator is positive because it equals \(P(E)>0\). Finally, summing the posterior formula over \(i\) gives

\sum_{i=1}^k P(C_i\mid E) = \frac{\sum_{i=1}^k P(E\mid C_i)P(C_i)} {\sum_{j=1}^k P(E\mid C_j)P(C_j)} =1.

Thus the denominator both accounts for every possible class that could have produced the evidence and normalizes the posterior probabilities. If a class has prior probability zero, its posterior is zero whenever \(P(E)>0\); it can simply be omitted from the formula. If \(P(E)=0\), conditioning on \(E\) is not defined by this event-based formula.

Key takeaway: The unnormalized posterior weight for class \(C_i\) is \(P(E\mid C_i)P(C_i)\), not just \(P(E\mid C_i)\). Divide each weight by the sum of all class weights.

Worked Examples: Find the Class After Observing Evidence

Worked Example: A Positive Test and a Rare Condition

Suppose 1% of a population has a condition. A test is positive for 90% of people who have the condition and for 5% of people who do not. Given a positive result, find the probability that a person has the condition.

Let \(D\) denote having the condition and \(+\) denote a positive test. The relevant probabilities are the prior \(P(D)=0.01\), the likelihood \(P(+\mid D)=0.90\), and the false-positive likelihood \(P(+\mid D^c)=0.05\). The complement has prior \(P(D^c)=0.99\).

The prior-weighted likelihood for the condition class is

P(+\mid D)P(D)=0.90(0.01)=0.009.

The prior-weighted likelihood for the no-condition class is

P(+\mid D^c)P(D^c)=0.05(0.99)=0.0495.

The total probability of a positive result is \(0.009+0.0495=0.0585\). Hence

P(D\mid +) =\frac{0.009}{0.0585} =\frac{90}{585} =\frac{2}{13} \approx 0.1538.

So the probability is about 15.4%, not 90%. The 90% figure is \(P(+\mid D)\): it describes the chance of a positive result among people who have the condition. The question asks for \(P(D\mid +)\), the chance of the condition among people with a positive result. The large no-condition group contributes many false positives despite its lower positive rate.

Worked Example: Defects from Two Factories

Factory A produces 80% of a company's items and has a 1% defect rate. Factory B produces the remaining 20% and has a 4% defect rate. An item is found to be defective. Find the probability it came from Factory B.

Let \(A\) and \(B\) identify the factory, and let \(D\) denote a defect. The priors are \(P(A)=0.8\) and \(P(B)=0.2\). The likelihoods are \(P(D\mid A)=0.01\) and \(P(D\mid B)=0.04\). The prior-weighted likelihoods are

P(D\mid A)P(A)=0.01(0.8)=0.008,\qquad P(D\mid B)P(B)=0.04(0.2)=0.008.

Thus \(P(D)=0.008+0.008=0.016\), and Bayes' rule gives

P(B\mid D)=\frac{0.008}{0.016}=0.5.

Although Factory B has four times the defect rate, it produces only one-fifth of the items. Those prior production shares matter: in this example the two factories contribute equal numbers of defective items in probability terms. Normalizing the defect rates alone would instead give \(0.04/(0.01+0.04)=0.8\), which is not the requested posterior.

Worked Example: Three Possible Sources of a Signal

A signal may come from one of three sources, \(C_1,C_2,C_3\), with prior probabilities \(0.5, 0.3, 0.2\), respectively. The probability of observing signal \(S\) is \(0.2\) from \(C_1\), \(0.5\) from \(C_2\), and \(0.1\) from \(C_3\). Find the posterior probability of each source given \(S\).

First multiply each likelihood by its prior:

\begin{aligned} w_1&=P(S\mid C_1)P(C_1)=0.2(0.5)=0.10,\\ w_2&=P(S\mid C_2)P(C_2)=0.5(0.3)=0.15,\\ w_3&=P(S\mid C_3)P(C_3)=0.1(0.2)=0.02. \end{aligned}

The total probability of the signal is \(0.10+0.15+0.02=0.27\). Divide each weight by \(0.27\):

P(C_1\mid S)=\frac{10}{27},\qquad P(C_2\mid S)=\frac{15}{27}=\frac{5}{9},\qquad P(C_3\mid S)=\frac{2}{27}.

These probabilities are nonnegative and sum to \((10+15+2)/27=1\), as required. Normalizing the likelihoods alone would give \(0.2/0.8=0.25\), \(0.5/0.8=0.625\), and \(0.1/0.8=0.125\). Those values also sum to one, but they answer a different calculation: they ignore the fact that the three source classes had unequal prior probabilities.

Posterior Odds and the Role of Base Rates

For two classes, Bayes' rule can also be written in odds form. This makes the influence of the prior especially visible. If \(P(C_1)>0\), \(P(C_2)>0\), and both evidence likelihoods are positive, then

\frac{P(C_1\mid E)}{P(C_2\mid E)} = \frac{P(E\mid C_1)}{P(E\mid C_2)} \frac{P(C_1)}{P(C_2)}.

To verify this, apply the two-class Bayes formula to both posterior probabilities. Their common denominator cancels:

\frac{P(E\mid C_1)P(C_1)/P(E)} {P(E\mid C_2)P(C_2)/P(E)} = \frac{P(E\mid C_1)}{P(E\mid C_2)} \frac{P(C_1)}{P(C_2)}.

The posterior odds equal the likelihood ratio multiplied by the prior odds. Evidence can favor one class, but its effect must be interpreted relative to how common the classes were before observing that evidence. In the test example, the positive-result likelihood ratio favoring \(D\) over \(D^c\) is \(0.90/0.05=18\). The prior odds are \(0.01/0.99=1/99\), so the posterior odds are \(18/99=2/11\). Converting these odds to a probability gives \((2/11)/(1+2/11)=2/13\), matching the direct calculation.

Common Mistakes and How to Check a Calculation

Reversing the conditional. The claim \(P(E\mid C)=0.9\) does not imply \(P(C\mid E)=0.9\). The first conditions on class membership; the second conditions on the evidence. Use Bayes' rule to change which event is conditioned on.
Dropping the prior. Using only \(P(E\mid C_i)\) treats the classes as though their prior weights were equal. Multiply each likelihood by its own \(P(C_i)\) before comparing or normalizing.
Forgetting the denominator. The products \(P(E\mid C_i)P(C_i)\) are joint-probability weights, not usually posterior probabilities. Divide them by their sum to obtain probabilities conditional on the observed evidence.
Normalizing the wrong quantities. Likelihoods may not sum to one across classes; even if they can be divided by their sum, that does not generally give the posterior. The normalizing constant is the sum of the prior-weighted likelihoods.
Failing to check the result. Every posterior must be between zero and one, and the posteriors over a complete class partition must sum to one. These checks can catch arithmetic mistakes, although they cannot by themselves catch a reversed conditional or omitted prior.

A useful calculation layout is to write one row per class with its prior, evidence likelihood, and their product. Sum the products for \(P(E)\), then divide each product by that sum. This keeps the direction of conditioning visible and makes the normalization auditable.

Name the observed evidence.
Write \(E\) for what has been observed and \(C_i\) for each possible class.

Record priors and likelihoods separately.
For every class, identify \(P(C_i)\) and \(P(E\mid C_i)\); do not exchange their meanings.

Weight and normalize.
Compute \(P(E\mid C_i)P(C_i)\), sum across classes, and divide each weight by that sum.

Check Your Understanding

For each question, keep the direction of conditioning explicit and check the posterior probabilities.

A condition has prevalence \(0.02\). A test is positive with probability \(0.8\) for people with the condition and \(0.1\) for those without it. What is the probability of the condition given a positive result?
In the three-source example, which source has the largest prior probability, and which has the largest posterior probability after observing \(S\)? Explain why these need not be the same.
Two equally common classes have evidence likelihoods \(0.3\) and \(0.6\). Find the posterior probabilities after observing the evidence.
Explain why \(P(E\mid C)=0.95\) alone is insufficient to determine \(P(C\mid E)\).
Suppose prior-weighted likelihoods for three classes are \(0.12, 0.08,\) and \(0\). Find the normalized posterior probabilities and state what the zero weight means.