Bayes Rule for Finite Latent Classes

From Class Priors to Posterior Probabilities

A latent class is a group membership that is not directly observed, even though the class affects what we observe. A patient may or may not have a disease; a manufactured item may come from one of several production lines; an observation may come from one of several mixture components. Before seeing the data, we assign probabilities to the possible classes. After observing evidence, we update those probabilities.

The calculation is simple to state but requires careful normalization. Each class contributes its prior probability multiplied by the probability of the evidence within that class. These contributions are then divided by their sum. The result is a set of posterior probabilities that accounts both for how common each class was initially and for how compatible the observation is with that class.

Definition: Let $C_1,\ldots,C_K$ be a finite partition of the sample space: the events are pairwise disjoint and their union is the whole sample space. The class prior for $C_k$ is $\pi_k=P(C_k)$. For an observed event $A$, the class-conditional evidence probability is $q_k=P(A\mid C_k)$, defined when $\pi_k>0$. The posterior class probability is $P(C_k\mid A)$, provided $P(A)>0$.

The priors satisfy $\pi_k\geq 0$ and $\sum_{k=1}^K\pi_k=1$. If $\pi_k=0$, conditional probability given $C_k$ is not defined by the elementary ratio formula, because it would require division by $P(C_k)=0$. We therefore do not assign a value to $q_k$ for such a class. Its posterior probability can still be determined: if the evidence has positive probability, that class remains a probability-zero possibility.

Bayes’ Rule for a Finite Class Partition

Write $I_+=\{j:\pi_j>0\}$ for the indices of positive-prior classes. The evidence $A$ must have positive probability for its posterior to be defined. For a positive-prior class, the posterior is its prior-weighted evidence probability divided by the total prior-weighted evidence probability.

Theorem (Bayes’ rule for finite classes): Suppose $C_1,\ldots,C_K$ form a finite partition, and let $A$ satisfy $P(A)>0$. For each $j\in I_+$, define $q_j=P(A\mid C_j)$. Then $$ P(C_k\mid A)= \begin{cases} \displaystyle\frac{\pi_k q_k}{\sum_{j\in I_+}\pi_j q_j}, & \pi_k>0,\\[1.2ex] 0, & \pi_k=0. \end{cases} $$ The denominator is positive and equals $P(A)$.

Proof. If $\pi_k=0$, then $P(C_k\cap A)\leq P(C_k)=0$, so $P(C_k\cap A)=0$. Since $P(A)>0$, it follows that $$ P(C_k\mid A)=\frac{P(C_k\cap A)}{P(A)}=0. $$ For a positive-prior class $k$, conditional probability gives $P(C_k\cap A)=P(A\mid C_k)P(C_k)=q_k\pi_k$. The partition property allows us to decompose $A$ into its disjoint intersections with all the classes. The zero-prior intersections have probability zero, so $$ P(A)=\sum_{j=1}^K P(A\cap C_j) =\sum_{j\in I_+}P(A\cap C_j) =\sum_{j\in I_+}\pi_jq_j. $$ This quantity is positive by the assumption $P(A)>0$. Dividing $P(C_k\cap A)=\pi_kq_k$ by $P(A)$ proves the formula for every positive-prior class. The zero-prior case was handled separately. $\square$

The denominator is not an additional unknown: it is the probability of observing $A$ under the full class model. It is also the normalization that makes the posterior probabilities sum to one. Indeed, summing the positive-prior posteriors gives $\sum_{j\in I_+}\pi_jq_j/P(A)=1$; zero-prior classes add zero.

Worked Example: A Positive Diagnostic Test

Suppose a disease has prevalence $0.02$. A test is positive with probability $0.90$ for a person with the disease and with probability $0.05$ for a person without it. Given a positive test, find the probability that the person has the disease.

Let $D$ denote disease and $D^c$ no disease, and let $A$ denote a positive test. The two class priors are $P(D)=0.02$ and $P(D^c)=0.98$. Both are positive, so the class-conditional probabilities are defined: $P(A\mid D)=0.90$ and $P(A\mid D^c)=0.05$.

The prior-weighted evidence contributions are $$ P(D)P(A\mid D)=0.02(0.90)=0.018, \qquad P(D^c)P(A\mid D^c)=0.98(0.05)=0.049. $$ Thus $P(A)=0.018+0.049=0.067$, and $$ P(D\mid A)=\frac{0.018}{0.067}=\frac{18}{67}\approx 0.2687. $$ The answer is about $26.9\%$, not $90\%$: $90\%$ is the probability of a positive test conditional on disease, whereas the question asks for the probability of disease conditional on a positive test. The complementary posterior is $P(D^c\mid A)=49/67$, and $18/67+49/67=1$, as required.

Posterior Odds and Evidence Ratios

For two classes with positive prior probabilities, comparing their posterior probabilities removes the common normalizing denominator. This gives a useful odds form of Bayes’ rule. It applies when the observed evidence has positive conditional probability under both classes; if one class assigns probability zero to the evidence, its posterior is zero and the ratio must be handled accordingly.

Theorem (posterior odds form): If $\pi_k>0$, $\pi_\ell>0$, and $q_k,q_\ell>0$, then $$ \frac{P(C_k\mid A)}{P(C_\ell\mid A)} = \frac{\pi_k}{\pi_\ell}\frac{q_k}{q_\ell}. $$

Proof. Apply the finite-class formula to each class. Both posterior probabilities have the same positive denominator, so it cancels: $$ \frac{P(C_k\mid A)}{P(C_\ell\mid A)} = \frac{\pi_kq_k/P(A)}{\pi_\ell q_\ell/P(A)} = \frac{\pi_k}{\pi_\ell}\frac{q_k}{q_\ell}. $$ The ratio $q_k/q_\ell$ is the evidence ratio: it measures how much more likely the observation is under $C_k$ than under $C_\ell$. The posterior odds equal the prior odds multiplied by this ratio. $\square$

Worked Example: Updating Probabilities Across Three Components

An item comes from one of three production lines, $C_1,C_2,C_3$, with prior probabilities $0.50,0.30,0.20$, respectively. Their defect probabilities are $0.01,0.04,0.10$. Given that an item is defective, find the posterior probability that it came from each line.

Let $A$ be the event that an item is defective. The prior-weighted contributions are $$ \pi_1q_1=0.50(0.01)=0.005,\quad \pi_2q_2=0.30(0.04)=0.012,\quad \pi_3q_3=0.20(0.10)=0.020. $$ Their sum is $P(A)=0.005+0.012+0.020=0.037$. Therefore, $$ P(C_1\mid A)=\frac{0.005}{0.037}=\frac{5}{37},\quad P(C_2\mid A)=\frac{0.012}{0.037}=\frac{12}{37},\quad P(C_3\mid A)=\frac{0.020}{0.037}=\frac{20}{37}. $$ These add to $(5+12+20)/37=1$. Although line 3 has only a $0.20$ prior probability, it accounts for $20/37$ of defective items under this model because its defect probability is comparatively high.

Finite Mixture Evidence in a Categorical Model

A finite mixture can be viewed as a latent class selected first, followed by an observation whose distribution depends on that class. For a particular observed category, the class-conditional probability of that category plays the role of $q_k$. To update class probabilities, use the probability of the observed category within each class as given; no additional factorization of the observation is needed.

Worked Example: Identifying a Latent Response Class

A study has three latent response classes with priors $0.40,0.35,0.25$. The reported outcome $R$ has class-conditional probabilities $P(R\mid C_1)=0.15$, $P(R\mid C_2)=0.50$, and $P(R\mid C_3)=0.20$. Find the posterior class probabilities after observing $R$.

The weighted contributions are $$ 0.40(0.15)=0.060,\qquad 0.35(0.50)=0.175,\qquad 0.25(0.20)=0.050. $$ The probability of observing $R$ is their sum, $0.060+0.175+0.050=0.285$. Normalizing gives $$ P(C_1\mid R)=\frac{0.060}{0.285}=\frac{4}{19},\quad P(C_2\mid R)=\frac{0.175}{0.285}=\frac{35}{57},\quad P(C_3\mid R)=\frac{0.050}{0.285}=\frac{10}{57}. $$ To check the normalization exactly, $4/19=12/57$, and $(12+35+10)/57=1$. Class 2 becomes the most probable class after observing $R$, because $R$ is most likely in that class.

Common Mistakes and Checks

A frequent error is reversing the conditioning: using $P(A\mid C_k)$ as though it were $P(C_k\mid A)$. They answer different questions. The former describes the chance of the evidence within a known class; the latter describes class membership after seeing the evidence. Bayes’ rule connects them but does not make them interchangeable.

Another error is to normalize the class-conditional probabilities $q_k$ directly. In general, the $q_k$ values do not sum to one across classes. The quantities to normalize are $\pi_kq_k$, because classes with larger prior probability contribute more often before the evidence is observed. A reliable calculation follows three checks:

Confirm that the class priors sum to one and identify which are positive.
Multiply each positive prior by its class-conditional evidence probability; do not define or use a conditional probability for a zero-prior class.
Check that the evidence denominator is positive and that the posterior probabilities sum to one.

If $P(A)=0$, the conditional probability $P(C_k\mid A)$ is undefined, so the formula cannot produce a posterior for that observation. If a class has zero prior probability while $P(A)>0$, its posterior is zero, regardless of any informal claim about how likely $A$ might be in that class. The prior assigns probability zero to the class, and the event of belonging to it remains probability zero after conditioning on positive-probability evidence.

Key takeaway: For finite latent classes, posterior probability is proportional to prior probability times class-conditional evidence probability. Normalize those products over positive-prior classes, verify that the evidence has positive probability, and assign posterior zero separately to every zero-prior class.

Check Your Understanding

Use the finite-class formula carefully, distinguishing prior probabilities from conditional evidence probabilities.

Two classes have priors $0.7$ and $0.3$, and the observed event has conditional probabilities $0.2$ and $0.8$. Calculate both posterior probabilities and check their sum.
Why is $P(A\mid C_k)$ undefined by the elementary conditional-probability ratio when $P(C_k)=0$? If $P(A)>0$, what is $P(C_k\mid A)$?
State the condition under which posterior odds for two positive-prior classes equal prior odds multiplied by an evidence ratio.
In the diagnostic example, identify which probability is $0.90$ and which probability is $18/67$. Explain their different conditioning directions.
If every positive-prior class assigns conditional probability zero to $A$, what does this imply about $P(A)$ and whether a posterior given $A$ is defined?